What I am doing now, May 2024

2021-05-04

See previous post. Time flies: I have joined and left Lazada in the meantime. Right now I am working as AI Scientist @ Shanda.

Bootstrapping Recommendation Service

The tech stack is simple:

• pairec from Aliyun for backend service. Not a fan honestly, as we are using maybe less than 5% of the codebase; the remaining 95% just complicate things and slow the team down.
• AWS Postgres for offline batch processing/online feature store. A single Flink job processing app tracking event from a Kafka topic, and appends to a user log table. Everything else live on the DB.
• DBT for managing SQL task and k8s-cronjob for scheduling batch task.

Pairec lacks A LOT OF things for the stack, mostly because the stack is on AWS, not Aliyun. If time allows, plan to add:

• Config/Secret live reloading. Plan to adopt koanf, a lighter version of viper.
• A/B testing or feature flag. The challenge here is locality of logic to reduce complexity, which I may write a note on. Growthbook uses feature flag, which may work out.
• ML model serving. Not a priority right now, because no user == no data == no machine learning. Open Inference Protocol seems exactly what we need.

Getting Into Go

The transition from Python feels quite natural, as I was comfortable with TypeDict, pydantic model and type annotation anyway. The aha moment for me, was treating golang == C with sane syntax + async runtime (goroutine and channel) + interface, instead of another new language. I also like that, Golang treats Makefile and SQL as first class citizen, instead of reinvents the wheel. The only thing I miss from Python are sum type and list comprehension.

This is how many lines of golang I have written since the switch:

# use this SO answer: https://stackoverflow.com/a/7010890
added lines: 4968, removed lines: 4345, total lines: 623