Data engineering focuses on building systems that collect, store, and process large amounts of data. It's an essential part of making data accessible for analysis and decision-making. SQL is a key tool for data engineers, helping them manage databases and create efficient pipelines for data movement and transformation. Below is a collection of articles that dive into the role of SQL in data engineering. These articles cover essential concepts, practical tips, and exercises to help you understand how to use SQL in data engineering tasks, whether you're designing a database, managing data pipelines, or optimizing performance. Perfect for both beginners and experienced professionals, these resources will help you build and refine your data engineering skills with SQL. 10 Oct 2024 Radu Gheorghiu Building Analytical Data Pipelines with SQL If you’re trying to build an analytical data pipeline, then SQL is the perfect tool for the job. It will help your organization build a data analytics foundation that turns data into business value. Why should you care about building data pipelines with SQL? This might sound like a technical challenge, but a strong data pipeline is one of the most essential tools for turning raw data into actionable insights. Read more 26 Mar 2024 Gustavo du Mortier How to Become a Big Data Engineer What is a Big Data engineer, and how is their skill set different from a data engineer? In this article, we explore the tools and platforms you’ll need to master as a Big Data engineer. To move from being a regular data engineer to a Big Data engineer, you need to acquire several new skills and learn to use several new tools. The good news is that Big Data still lets you use your good old SQL skills to manipulate and get information from data repositories. Read more 9 Jan 2024 Alexandre Bruffa Using SQL in Data Warehousing SQL is a key player in organizing and analyzing data in data warehouses. In this article, I'll introduce you to the basics of SQL in data warehousing. We’ll learn how this powerful language helps manage large datasets, making your work easier and more productive. Whether you're a student diving into the world of technology or an experienced specialist navigating the domain of data, understanding the concept of data warehousing is fundamental. Read more 15 Dec 2022 Maria Durkin Want to Get a Data Engineering Job? Learn SQL! So, you want to be a data engineer, but you don't know how to get there. In this article, we’ll help you understand the skills you'll need for different types of data engineering jobs. You’ll also find some data confirming that data engineer jobs are a good career choice. Data engineering and various types of data engineer jobs have become very popular in recent years. And for good reason: Next to data science, data engineering is one of the fastest-growing branches of IT. Read more 29 Nov 2022 Ignacio L. Bisso How to Choose Data Types for SQL Table Columns Databases store data values of various kinds. For example, you may store the name of a product as a text string, the price of a product as a number, or the date a product is sold as a date value. These values – the product name, the price, and the sale date – are stored as different data types. In this article, we explain several SQL data types with real examples. Read more 8 Feb 2022 Himanshu Kathuria What Is the Future of the Data Engineer? If you are wondering about the future of data engineering as a career and whether it is worth becoming a data engineer, then this article is for you. Here, I cover how lucrative the prospects are for data engineering and how you can get started and thrive in the domain. “Data” is probably one of the most used terms in our everyday business language today. A data engineer plays a pivotal role in developing the infrastructure required for data-related actions. Read more 12 Oct 2021 Andrew Bone Numeric vs. Decimal Data Types in SQL In standard SQL, there is a slight difference between the NUMERIC and DECIMAL data types. However, in some variants of SQL, there is no difference! This article will clear up any confusion. In this article, I’ll explain the differences and similarities between SQL’s standard NUMERIC and DECIMAL data types. Then I’ll discuss how these data types work in common SQL dialects. We’ve already posted a more general article on SQL data types; this time, we’ll drill down further on just the DECIMAL and NUMERIC data types. Read more 29 Sep 2021 Martyna Sławińska Why Use Primary Keys and Foreign Keys in Databases? Relational databases organize data in a standardized manner. All data is entered into tables, the so-called relations. The tables are linked with one another to create connections between the data. The backbone of any relational database is the primary and foreign keys. Read along to find out why they are so important. In this article, we focus on the reasons why relational databases use primary and foreign keys. First, we review what primary and foreign keys are. Read more 24 Sep 2021 Kateryna Koidan 7 Things Every Data Engineer Should Know People generate massive amounts of data every day. To get insights from data, organizations need to capture and process them efficiently. That is when data engineers are called up. In this article, I’ll discuss the data engineering role and the skill set necessary to succeed in the role. As the world generates more and more data every year, the IT industry creates new roles to deal with it. These roles include data analysts, data scientists, machine learning engineers, and data engineers. Read more 12 Aug 2021 Martyna Sławińska An Overview of MS SQL Server Data Types SQL Server data types define what can be stored in a column, local variable, expression, or parameter. It is essential to pick the right data type. Ultimately. your choice of data types affects the whole database. Read on to learn about all of the data types available in MS SQL Server. In this article, we’ll cover numerical, text, and date and time data type categories in detail. We’ll go through their syntax, storage size, and typical use cases. Read more 7 Jul 2021 Martyna Sławińska Primary Key vs. Unique Key: Explaining the Differences When designing a database, we often need to decide between defining a primary key vs. a unique key. Both of them are crucial during the design and further phases of a database. They allow us to uniquely identify each row, ensure the uniqueness of values in the column(s), and more. Let’s take a deep dive to learn more. This article will go through the primary and unique keys, their functions, and features. Read more 20 May 2021 Martyna Sławińska What’s the Difference Between UNIQUE and DISTINCT in SQL? We use the words unique and distinct interchangeably in our day-to-day life. However, these words play very different roles in SQL. Let’s find out what is so different about them and how we can use them in SQL. Despite the words unique and distinct being synonyms, they cover very different things in SQL. The UNIQUE keyword in SQL plays the role of a database constraint; it ensures there are no duplicate values stored in a particular column or a set of columns. Read more 29 Apr 2021 Ignacio L. Bisso Why Do You Need a Primary Key in a Database Table? What is a primary key and why is it so important in SQL? Find out in this article. Primary keys are a central element in SQL databases. They provide a unique value that can identify a specific row in a table, which opens up a lot of other benefits. This article covers what a primary key is, how it works, and how to use it in an SQL database. Read more 27 Apr 2021 Martyna Sławińska What Is the Benefit of Foreign Keys in SQL? What is a foreign key and why is it so important in relational databases? Learn all the details about foreign keys in this article. One of relational databases’ key features is the ability to link data stored in different tables. These links, called references, essentially act as connections between tables. They are created using the FOREIGN KEY constraint on table columns. In this article, we’ll learn what the FOREIGN KEY constraint does in SQL. Read more 23 Apr 2021 Andrew Bone What Is the Role of DISTINCT in SQL? The DISTINCT keyword in SQL plays an important role in removing duplicate values from your results. This article provides examples showing exactly how DISTINCT works. By default, SQL queries show all the returned rows, including duplicate rows, in the result set. The DISTINCT keyword in the SELECT clause is used to eliminate duplicate rows and display a unique list of values. In other words, the DISTINCT keyword retrieves unique values from a table. Read more 24 Mar 2021 Ignacio L. Bisso What Is a NOT NULL Constraint in SQL? SQL has several ways of dealing with NULLs. In this article, we’ll focus on the NOT NULL constraint and other NULL-related clauses. Understand this and you’ll take an important step in your SQL growth! Before we get straight into the NOT NULL constraint, let’s quickly define what NULLs and constraints are. NULL values are a central concept in SQL databases. The idea behind a NULL value is simple: a NULL value means that we do not have a known value for that field. Read more 4 Feb 2021 Martyna Sławińska What Is a Unique Constraint in SQL? To ensure the uniqueness of data in a database, we use the SQL UNIQUE constraint. In this article, we’ll discuss how, when, and why to implement it. The idea of a unique constraint is not unique to SQL; it’s a familiar concept in real life, although we probably call it something else! For example, think of ID numbers. There can be only one social security number (SSN) per person and each one must be unique. Read more 29 Jan 2021 Ignacio L. Bisso Data Types in SQL Read this article and learn the ABC of SQL data types. Data types are used to define the type of data stored in a database. However, there are related concepts that a good SQL developer should know: data type conversion, what operations are possible between different data types, date arithmetic, etc. SQL also provides functions that manipulate values of different data types (i.e. functions for string data, date and time data, etc. Read more 8 Jan 2021 Martyna Sławińska What Is Auto-Increment in SQL? In this article, we’ll learn the basics of the auto-increment feature of SQL: what it is and how to use it efficiently. One of the many features offered by SQL is auto-increment. It allows us to automatically generate values in a numeric column upon row insertion. You could think of it as an automatically supplied default value – the next number in a number sequence – that’s generated when the row is inserted. Read more 25 Dec 2020 Ignacio L. Bisso What Is a Database Index? Database indexes are critical to query speed and efficiency. Learn when, why, and how to use them – and when not to – in this post. Indexes are very important database objects; they optimize data access and improve database performance by helping the database execute SQL queries faster. Why not take this opportunity to learn how indexes work and how to create them? What Is a Database Index? Indexes are data structures that can increase a database’s efficiency in accessing tables. Read more 11 Dec 2020 Martyna Sławińska What is an SQL View? A view is a well-known feature in SQL. It allows you to create a virtual table based on an SQL query referring to other tables in the database. A view stores an SQL query that is executed whenever you refer to the view. This is a convenient way to get the desired data because it is easier to run a query stored in a view than to type a query from scratch. Read more 27 Nov 2020 Ignacio L. Bisso What Is a Foreign Key in SQL? What does a foreign key do in SQL? Why would you need one? Learn the ABCs of foreign keys in five minutes. Foreign keys are a central concept in SQL databases; they allow us to enforce data consistency. Usually they work with primary keys to connect two database tables, like a virtual bridge. All SQL developers need to know what SQL foreign keys are, how they work, what data values are allowed in them, and how they’re created. Read more 19 Nov 2020 Martyna Sławińska What Is a SQL Constraint? What are SQL constraints? When and why should you use them? We give examples of common constraints and demonstrate how to apply them. When you’re creating a relational database, you often want to impose certain restrictions on some columns. For example, a column containing a social security number should store only unique values; a column containing a date of birth should not be null, or left empty. These conditions can be ensured by using SQL constraints. Read more 11 Nov 2020 Zahin Rahman Top 7 Online Courses for Data Engineers This article summarizes the top online courses available for data engineers. We have picks suitable for beginners as well as intermediate learners. If you’re interested in database design and management, check these courses out! Most individuals who aspire to enter the realm of data aim for data scientist or data analyst roles. While these roles are indeed very rewarding because of their tangible links to customers and business direction, the role of data engineers is equally vital for businesses that operate in a data-rich environment. Read more 22 Oct 2020 Ignacio L. Bisso What is a Primary Key in SQL? Primary keys are an important concept in SQL databases. They provide a unique ID for every row in a database table. As an SQL developer, you should know what primary keys are, what data values are allowed in them, and how to create them. Read this article to learn the ABCs of primary keys in 10 minutes. How to Identify Rows in a Table Tables are the main objects in an SQL database, and as you probably know, tables store records or rows. Read more 20 Oct 2020 Adrian Więch What Is a DBMS? We generate vast quantities of data every day, and that data needs to be stored somehow. That’s where DBMSs come in handy. Find out what they are and how they relate to databases. If you go on a diet and simply want to keep track of your weight, you can probably use a piece of paper and a pencil. If you then want to chart your daily caloric intake, you’ll probably switch to something like a computer spreadsheet. Read more 7 Oct 2020 Marija Ilic How to Create Your First Table in SQL Creating a database table with SQL is one of the core skills you’ll need to work with data. And it’s easy to learn, so let’s get started! Imagine you’re analyzing data and want to store your results in a database table. Sure, you've done this a million times in Excel. But you’re not sure how to create a table with SQL. Or maybe you’ve noticed that data engineering is in high demand and you want to start learning its core concepts. Read more 18 Sep 2020 Marcin Koryszewski How Much Do Data Engineers Earn in 2020? Data engineering is a hot job right now. What does a data engineer do? And just how much do they earn? We’ve done the research for you. If you are reading this, you’re either starting to learn SQL or you already know some and feel it’s time to look around for a SQL job. Maybe you’re in the midst of tackling our Creating Database Structure track. And, let’s be honest, maybe you need some extra motivation to keep on you track. Read more 19 Aug 2020 Dorota Wdzięczna How to Create a Table in SQL Creating tables in databases is a very helpful skill, and not just for software engineers or database administrators. It allows you to design or change the structure of a database and store data that’s related to each other. In this article, you’ll learn what a database table is, who creates them, and how to use the syntax of the CREATE TABLE command. What Is a Database Table? A relational database is built of various structures like tables, views, procedures, and triggers. Read more 28 Jul 2020 Adrian Więch Who Is a Data Engineer? A new kind of job has recently emerged in the IT world: Data Engineer. At first sight, it may seem very similar to Data Analyst or Data Scientist positions. However, our article explains all the important differences. We present the skills, tools, and everyday tasks of Data Engineers. We also explain how you can get started with this career path. Thirty years ago, we typically used terms such as “Computer Scientist” when referring to anyone working with computers. Read more 10 Jul 2020 Adrian Więch What Is Data Engineering? Data Engineering is a fairly new term in IT. And it’s getting more and more attention. You may have heard about a few similar fields like data science, Big Data, and machine learning. This article explains the difference between these concepts and shows how they can be combined to analyze vast amounts of data. When computers first appeared, their storage capacity was very limited. Do you remember floppy disks? They were popular in the late 20th century and typically offered around 1. Read more 24 Jun 2020 Dorota Wdzięczna Comparing TRUNCATE TABLE, DELETE, and DROP TABLE in SQL What’s the difference between truncating, deleting, and dropping a table in SQL? Find out in this article. There are a lot of ways to delete data in SQL, including the DELETE, TRUNCATE TABLE and DROP TABLE commands. Which one should you use in a given situation? In this article, you’ll learn the syntax of each command in different database engines like MySQL, PostgreSQL, SQL Server, and Oracle. And you’ll understand the DROP TABLE vs. Read more 14 Aug 2018 Aldo Zelen SQL Indexing 101 Indexes are one of the most misused and misunderstood entities in physical database design. A good understanding of indexes and how they solve database performance problems is necessary for any database novice. In this article, we'll look at basic database indexes and their role in database development. To picture what an index is, consider a textbook. At the end of most textbooks is an index listing all the terms one can find in the text and the pages on which they appear. Read more 18 Oct 2017 Ignacio L. Bisso Learn SQL Views in 30 Minutes Views aren't complicated – if you've got half an hour, we'll get you started writing SQL queries using views! Let's start by answering the question "What is a view in SQL?'. A view is a database object (as is a table, an index, or a stored procedure). Like a table, you can query a view and extract the information in it. It can be used in the FROM clause of a SELECT, and you can reference view columns in clauses like SELECT, WHERE and GROUP BY among other clauses as well. Read more 17 Aug 2017 Francisco Claria Referential Constraints and Foreign Keys in MySQL Foreign keys and referential constraints allow you to set relationships between tables and modify some of the database engine’s actions. This beginner’s guide explains referential integrity and foreign key use in MySQL. One of the most important aspects of database usage is being able to trust the information you store. Database engines provide several features that help you maintain the quality of your data, like defining required columns as NOT NULL and setting an exact data type for each column. Read more
Data engineering focuses on building systems that collect, store, and process large amounts of data. It's an essential part of making data accessible for analysis and decision-making. SQL is a key tool for data engineers, helping them manage databases and create efficient pipelines for data movement and transformation. Below is a collection of articles that dive into the role of SQL in data engineering. These articles cover essential concepts, practical tips, and exercises to help you understand how to use SQL in data engineering tasks, whether you're designing a database, managing data pipelines, or optimizing performance. Perfect for both beginners and experienced professionals, these resources will help you build and refine your data engineering skills with SQL.