We kind of have a hybrid of our own stuff. We're using Control-m generally, but then we're making use of a cyclic lambda jjob that checks our database tables and SNS to check s3 file landing spots.I’m helping lead a transition to the cloud, and the leap is insane, more so because of petty stuff like not being given full IAM permission rights.
This is our first time using airflow and there’s a bit of a learning curve. We’ve finally got a handle of much of what we need to do to get started but damn.
There was like no help or guidance for all the frustrating ish we encountered.
Documentation is definietly important, so we can hand things off to someone else