Mastering Data Migration: Insights From Expert Arockia Nirmal

Learn about the importance of data migration and how to become an expert in this field yourself

Recently, I had a great talk with Arockia Nirmal, a data migration expert and founder of ZippyTec. In our interview, Arockia discusses his journey in data engineering, the significance of data migration, and insights on advancing in this field through his consultancy and coaching services.

@Image by Freepik

**Andreas:** Hey everybody, my name is Andreas from LearnDataEngineering.com, and today is all about data migration. My guest, Arockia Nirmal, is an expert in data migration with many years of experience. He's the founder of ZippyTec where he is doing consulting for companies in data migration and he's also doing coaching for students where you can learn that skill. Pay him a visit on LinkedIn and also check out his company page. So, hello Arockia.

**Arockia:** Hi Andreas!

**Andreas:** Nice to have you here. Maybe we start with you telling us a bit about yourself.

**Arockia:** Yeah sure. Actually, first of all, thank you very much Andreas for taking this time and having me with you. And also, thank you very much for your nice introduction. My name is Arockia Nirmal. I am a data engineer based in Germany. I live with my family near Karlsruhe. So basically, I started my career as a data access tester and developer back in the year 2012 in India. After that, I came to Germany for doing my Masters. Since then, I'm working in Germany as data engineer, database developer, DevOps consultant, and several other data-like consultant roles. At present, beginning January this year, I started my own consultancy called ZippyTec where I am offering consultancy and coaching services to individuals and other companies out there on data engineering and database migrations.

**Andreas:** You're specialized in database migration. What is that skill? What is database migration?

**Arockia:** Simply put, database migration is the process of transferring or moving data between different database systems. The database systems might be of the same type or different storage types and formats and so on. So, it actually involves moving schema data like tables or indexes and data to other databases while ensuring data integrity and minimal downtime.

**Andreas:** Why is this then such an important skill to have right now? Is this something super needed?

**Arockia:** Yeah, I can think of a couple of reasons from my experience basically. So, the first thing is business or like technology evolution. Companies often want to migrate databases when they upgrade their systems or change their business requirements or want to leverage new database technologies. For instance, a company might want to move from a traditional on-premises database to cloud solutions for better features, you know, security, and scalability, and so on. The second reason I see is something like data unification. So during mergers or acquisitions, companies often need to consolidate their database requiring seamless database migration to avoid disruptions to their business operations. So businesses with multiple databases might also want to consolidate their data into a single centralized system which is also known as single version of truth.

**Andreas:** So, you would say every company needs these people?

**Arockia:** Yeah, whichever company owns databases, they of course need some specialized person in database migrations.

**Andreas:** You have worked for some time now on this topic. Can you tell us a bit about your experience in that?

**Arockia:** Yeah, in one of my initial projects, I was made in charge of building a data warehouse from scratch. So, we have different sources of data, something like OLTP databases and like CSV files and Flat files. So, they asked me to build a data warehouse where the sales and marketing team can see this reports in business intelligence tools. So, what I did was, so this OLTP database was based on Microsoft SQL Server. I took SQL Server Integration Services, exactly. So, my first job was to build ETL jobs that take the data from OLTP databases and different data sources and populate this data warehouse. I actually automated the stuff over there and I also implemented testing, everything from scratch. So, that was my first data migration project if I remember correctly. That was like eight or nine years ago.

And recently, I did a database migration project for one of the big companies in Germany. So, what we did is we migrated their on-premises SQL Server to AWS Aurora MySQL. So, I used database migration services and schema conversion tools to accomplish this database migration process. The thing is in my past experience, in every job, whether I worked as database engineer or data engineer or database developer, there was this specific part of the job where I had to do database or data migrations.

**Andreas:** Are there then some core pillars to this that people should learn? Because as you said, it's a skill that sooner or later you're going to need.

**Arockia:** Actually, that's an interesting question. So, I can think of five key aspects. Something like foundational knowledge, then migration tools and technologies, security, and then testing and validation, and at last, I would say post-migration optimization. To make it much simpler, maybe we can think of an example, like we have a shop or we have a store maybe in Frankfurt, Germany.

So, we run this shop for more than two years now and over the recent days, we see our shop is getting traction. So, a lot of customers really like our shop and during the weekends, there is a lot of crowd nowadays that we could not be able to manage. So, what do we do in this case? As like management, we will plan and say okay, we want to build a new shop. To build a new shop, we need to choose a nice location. We need to choose like materials and so on to construct the building. So, this is kind of foundational knowledge, like understanding databases, data modeling, and I would say SQL.

And then once we have everything planned regarding materials, location, and so on, we need a strategy like how we are going to move the items from our current shop to the new store. We can do it step by step or we can transfer things, everything at one shot. We need to package things from the old shop and move it to the new store. We need to find some kind of, I would say, transportation service, maybe a very big truck and a safe truck. So, this can be related to migration tools and strategies.

There are different types of migration strategies, something like big bang migration, trickle migration, phased, and then zero downtime migration strategies. And then we have this ETL tools that do the job of packaging the items from the old database and putting it in the new database. And then, of course, we have this migration tools, those are like transportation, which really enable us to move our data very fast between the old and new database.

And of course, the next pillar or next key aspect, I would say, is security. During this database migration and after the migration, we need to ensure that no unauthorized personnel can access the data. Like in our example, any thieves or unauthorized person should not steal the items from our shop, from our new store. We need to ensure security.

And the next thing, the next aspect is testing and validation. We need to make sure that the count of items that we transferred between the old and the new store remains the same and also none of the items are broken or damaged.

And the final pillar would be optimization, like after post-migration. We need to optimize our new shop so that we can gain a lot of business value from it. So, I would say these are the core pillars of database migration.

**Andreas:** Okay, there is a lot that I have already heard of and that I'm also teaching to students. But it's not that easy, right?

**Arockia:** Yeah. From a bigger perspective, it seems to be a simple task like yeah, it's migration, just an ETL job, something like that. But I can tell you from my battle scars, sometimes we had major issues and we overcome it in a very difficult way after learning a lot of lessons. So yeah, it's very much critical for the business.

**Andreas:** So, for anybody who doesn't have that skill right now, what are the prospects of somebody getting that skill or having that specific skill of data migration?

**Arockia:** Yeah, I would say advancement in the field of data engineering or like data management, either it can be a full-time or freelancing job. So, database migration is a very, very important skill in these fields. And experienced professionals can receive really high salaries. And also, they could work for a variety of companies. As I told earlier, every company that relies on a database is a potential employer, from established enterprises to cutting-edge startups. And also, I have seen companies, they actually go to external consultancies or other experts for specifically doing this database migrations. And the internal teams, they will work on developing ETL jobs or like data pipelines. So, when one can show the potential employers that they are expert level knowledge in this database migration, of course, the employers are going to get attracted when they show something like a nice project about where they implemented this database migration, where they planned this, and also where they automated the testing strategies. Then it's going to be really attractive for the future employers.

**Andreas:** So, for future employers but also for advancing in the company that you already are because you are going to have more opportunities there.

**Arockia:** Yes, of course.

**Andreas:** So, with ZippyTec, you're doing this for companies, but let's talk a bit about actually teaching this to people. How are you teaching this to people?

**Arockia:** Yeah, so basically, my coaching program is more flexible and it spans like 6 weeks. So, in week 1 to 4, I want to concentrate on different aspects of data or database migration. For example, the first week, we are going to go through common challenges faced in database migrations and planning for a migration and all risk assessment and so on. And also, we are going through the case studies, like showing successful or failed database migration projects. So, the first week is more or less kind of theoretical, and we have some couple of hands-on going over the case studies. In week number two, we are going to concentrate on tools and technologies for database migration. For the sake of simplicity, for this coaching I took this AWS DMS service and the schema conversion tool. So, I will be having pre-recorded videos to show the learners what are these tools and how can they use it. And I will also give the learners something like Terraform templates so that they can create their own infrastructure for this project, maybe in AWS, and they can also use other cloud providers. Of course, I'm going to support them. And in week three, we are going to do a demo project on schema and data migrations. And I'm going to teach them the monitoring strategies, like error debugging mechanisms and so on. The fourth week, I'm going to completely dedicate this for testing and test automation because this is one of the critical parts I saw in my previous experiences that companies overlook and where we didn't have a dedicated automated testing architecture. So, we are going to do some testing and we are going to do some automations using Robot Framework or pytest and Great Expectations Python libraries and so on. And week five and six is completely dedicated for Capstone project. So, here the learners can bring their own topic or their own problem which they face currently, I mean the database migration problem. So, they can work on this Capstone project for a couple of weeks. And one more thing is, every week, I will be having this one-on-one session with the learners where I will be personally monitoring them and their current progress, and I'm going to help them with their challenges. And of course, my coaching program is completely flexible. The learners can do their demo projects or their work at their free time. So, here basically, I would expect something like 10 to 15 hours of work per week to successfully complete this coaching program.

**Andreas:** What are the prerequisites that people should have for this? Because I guess that's a bit of the same than doing this for a company or in a company.

**Arockia:** So, for this coaching program, I would expect basic knowledge in database development or administration. Of course, proficiency with SQL scripts and basic knowledge in Python would be nice since we want to do this test automation. And also, basic understanding of Git and infrastructure as code, something like Terraform.

**Andreas:** Would you say this is something anybody needs to do or to get into this topic?

**Arockia:** Yeah, yeah. As I told, they don't need to be proficient on all this topics. I'm going to help them along the way. But if they could know already something like what is infrastructure as code, what are Python libraries, then it would really help me to fast-track their learning experience. Of course, I can make this coaching program flexible for individual persons. But for me to make them gain expertise in a faster and in a smooth way, I would expect them to have these prerequisites.

**Andreas:** How far do you think would they be able to go with this program?

**Arockia:** When designing this coaching program, my goal was to equip learners with the skills and knowledge to become proficient database migration specialists. So, by the end of this coaching program, one will be able to grasp important concepts of database migration and also migration strategies and security best practices and also utilize database migration tools and also testing tools to automate and streamline their everyday work. And last but not least, tackle the real-world migration projects.

**Andreas:** That sounds awesome. So, where can people find you? Where should they go?

**Arockia:** They can find this coaching program on my company's website, it's called ZippyTec.de - and still, we have another easier way in my LinkedIn profile. Under the featured section, I pinned the link to the coaching program. So, the learners can buy this coaching program anytime. We can even start immediately.

**Andreas:** Very cool. Yeah, I'm going to link this below as well, your LinkedIn and ZippyTec homepage. Arockia, thanks for being here. It was great talking to you. Good luck on your endeavors.

**Arockia:** Same here, Andreas. And thank you very much. I really appreciate your time for me. Good luck to you too.

Start Your Data Migration Journey Now

Already on fire to get into data migration yourself? Then get in touch with Arockia on LinkedIn and also check out his company page. With his well-founded coaching program you can equip yourself with the knowledge and skills needed to execute successful database migrations.

You can also watch the complete interview recording with Arockia and me on YouTube.

By the way, Arockia is part of my newly established Turbo Mentors program that I offer in my Academy. Here I train the next generation of online coaches. You want to be part of it? Learn more here!

🍀

Read my free 80+ pages Data Engineering Cookbook on GitHub: Read the Cookbook

Follow me on: LinkedIn | Instagram | X (Twitter) | YouTube |

Learn Data Engineering at my Data Engineering Academy, trusted by over 1,500 students 💪: Click here to learn more