Sunday, August 30, 2020

Invoking AppStore Connect API via Command Line

Sometime back Apple had opened up the AppStore Connect API for its developers. The API is a bunch of RESTful service which can be used to customise and automate our workflows. It helps us automate tasks across developer tools, such as App Store Connect, Xcode, and Certificates, Identifiers & Profiles, to give us greater flexibility and increased efficiency in our workflows. We could use it for development, beta testing, managing app metadata, reporting etc.

It can be very helpful to invoke the AppStore Connect API via command line. I use it primarily to check the structure of the response object. Although the documentation of the AppStore Connect API is good, but there is nothing as good as seeing the response object with real values in it.

In this post we will see how can we invoke the AppStore Connect API from the command line.

How To Do It? 

To invoke the AppStore Connect API via command line we will follow these steps.

  • Create API Key from the App Store Connect Web Portal
  • Create JWT JSON Token for Accessing API using XCToken
  • Invoking the API from command line using curl.

Create API Key 

  • To generate an API Key you will need to login to iTunes Connect portal. 
  • Navigate to the "Users and Access" section


  • In there, navigate to the "Keys" tab
  • Choose the "Key Type" as "AppStore Connect API"
  • Make a note of the "Issuer Id" we will be using this later on to generate the JWT JSON Token.


  • Generate a New API Key, by clicking the "+" icon. This will open up a dialog which ask you to name the key and select the Access type. I am going to choose "Developer" for this post.


  • This will generate the API key, make a note of the generated Key Id we will be using it later to generated the JWT JSON Token.
  • iTunesConnect will give you an opportunity to download the generated key only once. So please make sure you download it and keep it safe.


  • This completes the steps necessary to generate an API Key.

Create JWT JSON Token

To Generate the JWT JSON Token we are going to use a command line utility called XCToken. It can generate on-demand JWT tokens forAppStore Connect API. To install the utility just run the following command.

gem install xctoken
To generate the JWT JSON Token this utility expects three environment variables.
  • ISSUER_ID = This is the issuer id which we have noted in the earlier step from the iTunes Connect page.
  • KEY_DIR = The full directory path where the API key was downloaded from iTunes Connect
  • KEY_ID = The key id of the newly generated API key.
Once these environment variables are set up the XCToken is ready to generate the JWT JSON Token. Here is the sample script that will generate the token.

export ISSUER_ID=ABCD
export KEY_DIR=~/Downloads/
export KEY_ID=ABCD1234
xctoken  generate
This will spit out the token on the console, make a note of this token, we will be using it to invoke the AppStore Connect API.

Invoking the API

Now that we have generated a new JWT JSON Token, we are ready to invoke the AppStore Connect API. Here you can find various endpoints and their documentation. For e.g. to get a list of all your apps, use the following command.

curl  https://api.appstoreconnect.apple.com/v1/apps --Header "Authorization: Bearer <GENERATED TOKEN>"
Thats about all we need to do to invoke the AppStore Connect API via command line!

Friday, July 31, 2020

How To Reduce the Disk Space Need for Amazon Redshift - Part 2

This post will conclude the, process of reducing the disk space need for Amazon Redshift. If you haven't already read Part 1 in this 2 part series, I strongly recommend that you go read it now, I will wait!

Right, so now that we know why we are doing what we are doing, let's get straight to the point. How to actually do it?

How To?


We will use an open source library called Spectrify. It basically helps us with the following
  • Export a Redshift table to S3 (CSV)
  • Convert exported CSVs to Parquet files in parallel
  • Create the Spectrum table on your Redshift cluster
It basically performs all the steps needed to get our table offloaded from Redshift and then setup as a spectrum table on the redshift cluster. 

Since the entire process is network and CPU intensive, its advisable that we do it from an Amazon EC2 instance - t2.xlarge is highly recommended. Assuming that you have an Amazon EC2 instance, heres the script gets the job done.


The script is designed in such a way that it installs all the necessary dependencies required for its execution.

It basically performs the following steps:
  • Ensure that we have psql installed. Thats needed so that we can execute the commands on Redshift database. Install it if required.
  • Create the schema if required. It needs to be done only once. We will host our spectrum tables in spectrum schema.
  • Check if spectrify is installed. Install it with all the necessary dependencies, if required.
  • Export the data in the Redshift tables to CSV files on S3.
  • Convert the CSV files to Parquet format.
  • Moving the files to appropriate move path, so that we can support incremental exports.
  • Create the spectrum table in Redshift. Don't need to create the table again and again. Only do it once.
  • Truncate the Redshift table if required.

Once the script finishes running, you should have your spectrum table ready to be queried and used like any other Redshift table. It will have the same structure and data as the original Redshift table!

What about Incremental Offload?


Totally possible! We can run this script again and again on any table in Redshift. It will keep appending data to the spectrum table without overwriting the earlier data. If a table was offloaded earlier, running it again on the same table, will offload only new rows that got added after the last run.

I hope more and more people are able to use of this awesome feature and reduces the disk space need for Amazon Redshift cluster!

Tuesday, June 30, 2020

How To Reduce the Disk Space Need for Amazon Redshift - Part 1

At Makkajai, we use Amazon Redshift as our data warehouse. Redshift has got good features, but when it comes to providing disk space its a bit expensive. Especially, if you choose the DC kind of a cluster. For e.g. the dc2.large comes with only 160 GB storage. 

With any data warehouse, the data constantly keeps increasing and we always run into disk space related issues.

One of our cluster recently ran out of disk space. I started looking around for possible solutions to get around this problem.

Spend More Money

One very obvious alternative was to Scale the Amazon Redshift cluster horizontally.

This means adding a new Amazon Redshift nodes. While this solution is pretty easy, it comes with cost. An additional $140-$190 (approximately) per month + taxes. 

Also the data will keep piling up and we will have to keep adding new nodes to increase the cluster capacity. Hence, this approach becomes more and more expensive over time. 

Offload the data somewhere?

Theoretically, we could offload a lot of data to a cheaper storage (for example Amazon S3) and use that for querying data from Redshift. But we need to make sure that the data is still queryable from Amazon Redshift, just like we query any other table - that would be perfect. 

I looked around to see, if there is was a way to get this done with least amount of pain. Thankfully, we found Amazon Redshift Spectrum

It is build for specifically this use-case. 

... Being able to query data stored in S3 means that you can scale your compute and your storage independently, with the full power of the Redshift query model and all of the reporting and business intelligence tools at your disposal. Your queries can reference any combination of data stored in Redshift tables and in S3.

When you issue a query, Redshift rips it apart and generates a query plan that minimises the amount of S3 data that will be read, taking advantage of both column-oriented formats and data that is partitioned by date or another key.

But how much do we pay extra?

Spectrum pricing is based on the amount of data pulled from S3 during query processing and is charged at the rate of $5 per terabyte (you can save money by compressing your data and/or storing it in column-oriented form). You pay the usual charges to run your Redshift cluster and to store your data in S3, but there are no Spectrum charges when you are not running queries.

I think this paragraph summarises it pretty well.

What do we eventually get?

Using this approach we were able reduce the Disk Space needed for Amazon Redshift significantly. We were nearing 90-100% earlier, after offloading few bigger tables we are at 65-67% disk space usage.

In the next post, I will document the exact steps needed to get this setup in place. 

Sunday, May 31, 2020

Book Summary: It Doesn’t Have to Be Crazy at Work

Another book and another great read for people who want to reclaim a calm lifestyle and get more done with less.

These recent set of books, have made a big impact on how we run our company. The book presents a compelling argument to reclaim the calm lifestyle. It also shows us how we could to do it too. What work principles really matter and which ones are there because of legacy reasons. 

Book Summary

More and more people are spending more and more time at work, but out of the 60, 70, or 80 hours a week people are pouring into work, how many of those hours are really spent on the work itself? Many of the hours are wasted in meetings and doing non-essential discussions and distractions etc. The key isn't more hours but lesser bullshit. Less waste and lesser distractions!

One of the most important thing that I have realised after reading this book is that, "Your company is a product". As with every product, it constantly needs to evolve, iterate on what is working, make it better and throw away what is not working. I had never imagined, the company as a product. Just realising that company could be though as a product, brings newer perspective to things. Typically companies start with some process and then never bother to change them. Do we do that with any of our products? 

Next idea that I liked was, "Make it up as you go". I have to be hones here, none of the big one-year, five-year plans have ever worked. None, not a single one. In fact we have landed at completely different places. The book talks about keeping short-term plans only, typically lasting about 6 week. What to do after that? Take learnings from the previous execution and execute another short-term 6 week plan. 

The biggest impact this book makes when it talks about defending your time at work. Do not break your time into small chops of 15 minutes here, 20 minutes there, 10 minutes another meeting etc. Doing good work is hard enough and to do it right, you need to have long periods of undivided attention. 60 minutes could be broken in many different ways 1x60 or 2x30 or 4x15, the quality of hour we are after is 1x60. 

Plan for eventual response rather than immediate response. Lot of times we have a habit of pinging someone and expecting a response immediately. This needs to change, the other person might be in the middle of something important. If it isn't a crisis (which it isn't most of the time) there is no good reason to disturb the other person. Breaking the flow is a productivity killer. If you need something from someone, send them an email and they would respond whenever they see the email, may not be now, may not be in an hour but could be at the end of the day. If it's truly urgent, then use an old invention called a phone call.

Another incredible point the book makes is about "The trust battery". 

The trust battery is charged at 50% when people are first hired. Then every time you work with someone at the company, the trust battery between the two is either charged or discharged based on things like whether you deliver on what you promise. 
This is a powerful idea, it helps us in assessing work relationships with greater clarity. A low trust battery is at the core of many personal disputes at work. When the battery is drained, everything is wrong, everything is judged harshly.

Commitment and not consensus, someone in charge has to make the final call, even if others would prefer a different decision. Good decisions don't so much need consensus as they need commitment. Companies waste an enormous amount of time and energy trying to convince everyone to agree before moving forward on something. Instead they should allow everyone to be heard and then turn the decision over to one person to make the final call. It's their job to listen, consider, contemplate and decide.

Launch and learn, it's as simple as, if you want to know the truth about what you have built, you have to ship it! 

Conclusion 

These are some of the things that had the biggest impact on me. This by no means is a good summary of the book, I would encourage you to grab your copy and read it today. It has a lot more to offer!





Thursday, April 30, 2020

Book Summary: Remote: Office Not Required

COVID-19 has pushed a lot of companies to enable remote working from immediate effect. This meant, equipping their employees with a laptop and continuing the day to day activities in the same manner as before i.e. when they were co-located.

This is not only inefficient but it is also very strenuous on everyone. To better understand how to effectively run Remote teams, I decided to read the book Remote: Office Not Required written by David Heinemeier Hansson and Jason Fried. I decided to summarise the learnings from this book so that, more and more people could embrace Remote working in the right way.

Of course it goes without saying, this summary should only inspire you to go read the actual book, if you haven't read it already. And read it again, if you have read it before :D.

Book Summary

The book starts by asking a compelling question,
If you ask people where they go when they really need to get work done?
Very few of them will say The Office. Even if someone says its the office, then they usually add a qualifier such as,

  • "I go very earlier to the office before anyone gets in to get stuff done"
  • "I stay late at night after everyone's left"
  • "I sneak in on the weekend"
Why do people feel these way? It seems like the office during the day, has become the last place where people want to be, when they really want to get work done. 

I am sure we have faced this situation many times ourselves but have either ignored it or never tried to dig into the root cause of this problem. 

The Benefits

The core reason seems to be, meaningful work needs stretches of uninterrupted time to get done. But in today's offices it's impossible to get such long stretches of uninterrupted time. The ability to be alone with your thoughts is, in fact, one of the key advantages of working remotely.

Commuting to office is another time killer. This is the time which is least productive and a complete waste for the everyone. Remote working could eliminate commuting to and from the office which could end-up saving around 300-400 hrs, thats almost 38-50 working days in a year!

Technology has made significant progress, doing a Video call with screen sharing at the same time, is no longer a distant dream. It a reality and there are more than a few options that could do it.

Biology, some people like to work early in the morning and some like to be the night owl's. Giving people this flexibility, could get a lot more work done.

Companies broaden their horizons as they could hire the best talent available from anywhere in the world. Not just from their city or even from their home country. 

How To Implement It

Remote working doesn't mean you can't have an office. Remote work is about setting your team free to be the best it can be, wherever that might be. Some people may choose to come to office and work from there, while some others could work from a Cafe or Home Office or a Co-Working space!

Communication is the backbone of successful Remote teams. It's extremely essential for each and every team member to communicate effectively. Written communication becomes super important. Your team members may not be in the same city or country but its essential that team members have a meaningful time overlap between their working hours, 4-6 hrs seems to be the sweet spot.

It's great to physically meet your team members once in a while. It has lots of positive impact, we should aim at getting everyone at one location at least 2 to 3 times a year for 3-5 days. As important as it is to have the entire company get together, it’s also a great idea to occasionally do a sprint with a smaller group to finish a specific project. If the company must make a mad dash to meet a deadline—with the unreasonable hours and pressure that implies—it can be nice to slave through the ordeal together.


For new hires, its a good idea to spend 4-6 weeks at the office. This helps them absorb the company culture easily. They get a first hand opportunity to see how things work and how people communicate and get things done. Before making the final hiring decision, it's absolutely must have to meet them in person. This allows you to get a feel for their character. Are they polite? Do they show up on time? Are they fundamentally decent? Do they treat people well? What does the rest of the team think? You can tell a lot from a quick face-to-face.

Working remotely doesn't mean working from Home. It means working from a remote setup which is free of domestic distractions. The remote setup could be in your home office, or a cafe or a co-working place near by, it needs to have consistent internet connectivity, power backup and free from domestic distractions. Invest in getting a proper desk, a proper chair, and a proper screen - 27 inches in high resolution!

Move to async mode of communication. Not every question needs an answer immediately. There’s nothing more arrogant than taking up someone else’s time with a question you don’t need an answer to right now. Questions that can wait hours, are great candidates to put in an email. Questions that require answers in the next few minutes can go into an instant message. For crises that truly merit a sky-is-falling designation, you can use that old-fashioned invention called the telephone.

Remember, in Remote teams, the work they do, is the only metric that matters.

Have a place where teams could have water cooler conversations. We all need mindless breaks, and it helps if you spend some of them with your team. That’s where the virtual water cooler comes in.

To enable constant flow of information running through the office, we could use questions like “What have you been working on?” Everyone chimes in with a few lines about what they’ve done over the past week and what’s intended for the next week. It aims to make everyone feel like they’re in the same galley and not their own little rowboat.

It’s easy to be a manager when all you have to do is manage the chairs. Making sure that the little worker bees arrive by nine in the morning and giving them an extra star on their score card if they stay past six—this is how much of management has operated since forever. Don't do this!

Remove roadblocks, empower everyone to take decisions confidently. There should be no begging to spend money on needed equipment to get the work done, and there are no expense reports to fill out.

Conclusion 

The book has a lot more to offer, this post is only a poor attempt at summarising it. I hope that you are inspired to read the book. Do let me know how you liked it.

Tuesday, March 31, 2020

How to access data of Redshift cluster from PostgreSQL RDS

We were using Amazon Redshift as our data warehouse. It is a columnar-oriented petabyte-scale,  OLAP database suitable for analytical queries and applications.

There were many times when we wanted to pull data from Amazon Redshift and power our dashboards. This is basically using an OLAP database for OLTP use case - it was obviously not going to work very well.

Amazon RDS provides a feature called DBLink. DBLink enables us to access Amazon Redshift table directly from Amazon RDS. We don't need to explicitly copy over the data from Redshift to RDS, DBLink handles it for us and moves the data at the block level.

In this post, we will review the steps needed to setup the DBLink on Amazon RDS. This post assumes that you already have a running Amazon Redshift cluster and an Amazon RDS PostgreSQL instance.

Setting up DBLink
  • First we need to enable the DBLink extensions on the PostgreSQL RDS instance.
  • Next step is the main step which creates the link to the Amazon Redshift cluster and PostgreSQL RDS instance.
  • To achieve this we need to create the server link and the user mapping
  • Please note that all the values that are enclosed within <angle brackets> need to be replaced with actual values from your environment.
  • You might have to use the private IP address of your Redshift cluster. It totally depends on your setup.
  • Now that the DBLink is created, its time to query the Redshift cluster from the PostgreSQL to check if everything is setup correctly
  • Everything enclosed within $REDSHIFT$ is the actual query that runs on Redshift and results are sent back to PostgreSQL.
  • To create the cache of the Redshift data, we need to create a materialised view. 
  • Query the view just like you query any other view.
  • To refresh the view, use the following code

Thats about it, these are the simple steps using which you can access data of Redshift cluster from PostgreSQL RDS instance.

Saturday, February 29, 2020

Book Summary: Ikigai - The Japanese Secret to a Long and Happy Life

I truly believe everyone should have reading as one of their habits. It gives us great insights and widens our horizons. Reading takes you places and gives you new ideas. At Makkajai, we have been buying lots of books for office reading. 

We observed that even though we have lots of good books in office, not everyone is reading them or making an effort to read them. To push everyone to read more often, we came-up with a simple idea. Everyone has to write one pager book summary every month! 

It doesn't matter what book you read, it doesn't matter how thick or thin it is. All that matters is that everyone writes a book summary for the book they have read.

Very recently, I had read the book Ikigai: The Japanese Secret to a Long and Happy Life by Hector Garcia and Francesc Miralles. It goes without saying that, this is a pretty awesome book. To keep up with the tradition, I did write a short book summary this book. Turns my team members liked the summary hence, I decided to convert he book summary into a blog post in the hope that, more and more people will find this book interesting and worth reading.

Book Summary

This books demonstrates various topics related to Art of Living. Its the art of living and not art of existence or survival :). The book defines what is ikigai and the rules of ikigai. In Japanese, ikigai (生き甲斐) is a combination of letters that mean "life" and "to be worthwhile". Authors of the book try to unravel the secrets of long and happy life.

How do they do it? They conducted a total of one hundred interviews in Ogimi, Okinawa - Japan to try to understand the longevity secrets of centenarians and super-centenarians i.e. people who have lived for 100 and 110+ years!

The village of Ogimi in Japan has the most number of centenarians and super-centenarians in the world. Authors talk about various experiences while interviewing centenarians and super-centenarians.

So what did they find? 

Ikigai can be illustrated with this diagram very easily. It’s basically an intersection of “What you are good at”, “What you love”, “What the world needs” and “What you can be paid for”!

  • Ikigai roughly translates into “the happiness of always being busy.” 
  • We all have a certain passion inside us, a thing that we love to do and the thing that keeps us going. A unique talent that gives meaning to our days and drives us to share the best of ourselves until the very end. 
  • If we don’t know what this thing is yet, then our mission is to discover it.
  • Everyone’s ikigai is different, but one thing we have in common is that we are all searching for meaning. 
  • When we spend our days feeling connected to what is meaningful to us, we live more fully; when we lose the connection, we feel despair. 
  • Our ikigai is hidden deep inside each of us and finding it requires a patient search. 
  • Once we discover our ikigai, pursuing it and nurturing it every day will bring meaning to your life.
  • One thing that everyone with a clearly defined ikigai has in common is that, they pursue their passion no matter what.

What are the rules for Ikigai?
  • Stay active; don’t retire. Keep doing things that you love to do.
  • Take it slow. Don’t be in a hurry all the time. This is one of the most important thing to remember. Live life fully from today, right now, in this moment, do not wait for some obscure time in the future to do it.
  • Eat less. Only about 80% of your hunger. Obviously its not easy to figure out what is 80% of hunger, but the idea is to not fill up your stomach to 100% of its capacity :)
  • Surround yourself with good friends. If you are to live longer, you will need good company when you are old.
  • Get in shape, exercise and regularly move your body. At least try and do mild exercise 3-5 days in a week.
  • Smile and do it often. This is something that we all can do more of.
  • Reconnect with nature. Appreciate its beauty. Grow a garden in your home, go for a trek!
  • Give thanks and mean it.
  • Live in the moment. A lot of times we are either worried about the future or sad about the past. Don’t do it, Live in the moment.

Avoiding Stress & Existential Crisis

Many people seem older than they actually are. Why do you think that happens?
  • Research into the causes of premature ageing has shown that, stress has a lot to do with it.
  • Existential crisis, on the other hand, is typical of modern societies in which people do what they are told to do, or what others do, rather than what they want to do. Peer pressure, makes us do things that we don't really care about.
  • People often try to fill the gap between what is expected of them and what they want for themselves with economic power or physical pleasure, or by numbing their senses
  • Those who give up the things they love doing, lose their purpose in life.

Be in Flow

Flow is the mental state in which a person performing an activity is fully immersed in a feeling of energised focus, full involvement, and enjoyment in the process of the activity!
  • We have to focus on increasing the time we spend on activities that bring us to state of flow, rather than allowing ourselves to get caught up in activities that offer immediate pleasure.
  • Concentrating on one thing at a time may be the single most important factor in achieving flow. Too many times we try to do things in parallel and divide our attention. This is a sure shot recipe for not achieving the state of flow.
  • Our ability to turn routine tasks into moments of micro-flow, into something we enjoy, is key to our being happy.
  • The happiest people are not the ones who achieve the most. They are the ones who spend more time than others in a state of flow.

Slow Living
  • Being in a hurry is inversely proportional to quality of life. As the old saying goes, ‘Walk slowly and you’ll go far.’ 
  • When we leave urgency behind, life and time take on new meaning.
  • There is a huge difference between being in a rush all the time vs keeping yourselves busy with things that you love to do.
  • Never stop doing the things that you love to do!
  • Slow living doesn't mean that we stop being busy, it means we stop being in hurry!
And finally here are a few secrets of centenarians in their own words:

What should we do to live a long and happy life?
  • Don’t worry
  • Cultivate good habits
  • Nurture your friendships every day
  • Live an unhurried life
  • Be optimistic
  • Eat and sleep, and you’ll live a long time. 
  • Learn to relax.
  • Your mind and your body. You keep both busy, you’ll be here a long time.

Overall, I have learned a lot from this book and pretty sure that everyone can learn many things from it. Do read it and let me know how you liked it.
Have some Fun!