The importance of fundamental knowledge for software engineers

Nowadays, the whole software development industry is growing as a tree to different directions.

New technologies and frameworks constantly appear, a few become popular, but most of them die shortly. So how to decide which direction to take? What framework to learn? There are tremendous amount of online courses but study takes a lot of time as it is not possible without some hands-on practice (coding).

Passion (for beginners only)

First of all, it is not about money. My strong advice is DO NOT EVEN TRY to get into software development just because you have read about salaries in this industry. It might either make you happy if you love it or get you in depression if you do not.

General Computer Science Fundamentals

As a first step, computer science fundamentals and principles are the most important part. By these, I mean very general ones, which are relevant to any language or framework. Examples are:

• Data structures (e.g., canonic lists, trees, arrays, hash tables, queues, stacks, etc.)
• Basic algorithms on these data structures (e.g., searching, sorting, calculating hash code, etc.)
• Analysis of algorithms (time and memory complexity)
• etc..

Studying means not only reading about but trying some of them hands-on with any convenient language.

Why is it so important? Because there is no miracle. All complex software products are based on these fundamental principles and their variations. If you are not familiar with them, many higher-level things you meet is a "magic" for you. There should be NO MAGIC at all.

Example 1

During SQL query optimization, you see in query execution plan that database engine uses "hash join". Using knowledge of what "hash" means and how "join" is implemented, it might be clear for you that this type ofjoin is inappropriate in this particular query and this is the point to improve the query. Without this knowledge, you just blind why your query takes 15 secs instead of obviously expected 0.02 secs.

Example 2

Let's consider two following code snippets (C#). Semantically, they are the same, but what is happening inside?

1  using(var db = new MyDbContext())
2  {
3    foreach (var invoice in db.Invoices)
4    {
5      if (invoice.InvoiceDate > new DateTime(2015, 01, 01))
6      {
7        // do some work
8      }
9    }
10 }

1 using(var db = new MyDbContext())
2 {
3  foreach (var invoice in db.Invoices.Where(i => i.InvoiceDate > new DateTime(2015, 01, 01)))
4  {
5    // do some work
6  }
7 }

Both snippets work fine on your test environment. But why is the first so slow on production? If you do not know what is happening under the cover, it might be difficult for you to spot the problem. Here is the reason:

Entity Framework DbContext builds expression trees (surprise, surprise - fundamentals!) from expressionsdb.Invoices and db.Invoices.Where(...) respectively. Then, it builds SQL query from the expression tree and executes it against a database when the first time your code needs the result, like for foreach loop in our case.

So for the first snippet, the steps will be:

• Construct query "SELECT * FROM Invoice" at line 3
• Execute the query at line 3 and load all the invoices from database (assume thousands of them on prod)
• Loop with foreach through the invoices and do something for those that pass the condition check at line 5

For the second snippet, the steps will be:

• Construct query "SELECT * FROM Invoice WHERE InvoiceDate > '2015-01-01'" at line 3
• Execute the query at line 3 against the database and get only those invoices that match the condition.
• Load only filtered invoices from the database (it is obviously way faster than to load all of them)
• Loop with foreach through this filtered subset of invoices and do something at line 5

The example is trivial, but my point is fundamental knowledge gives you clear understanding of what is happening under the cover and drives you to effective code.

Moving further, during selection of a particular framework/technology or product for your real-life project, fundamental knowledge of what is happening inside helps make the best choice.

Programming Paradigms and Methodologies Fundamentals

The widely used programming paradigms are as follows:

• Imperative programming
• Object-oriented programming
• Declarative programming
• Functional programming

It is very important to clearly understand each of them and distinquish them in mind. Especially from viewpoint that many popular programming languages (C#, Java, Scala, Swift, etc.) adopt all of them (multiparadigm).

I would strongly recommend to take the deepest look at declarative the functional programming (to be honest, it is "Back to the Future" as soon as most of the concepts were invented decades ago). It significantly improves your mindset towards better abstract and structured thinking and as a result to better software developer. (Again, there are many good online courses and books available, like this course). As a light example, the functional programming with its key concept of usage immutable structures only are almost the only way to build scalable parallel programs.

Selected Technology Fundamentals

The next step is all about fundamentals of selected technology stack. Examples might be:

1. Java, Scala, JVM, bytecode, etc.
2. C#, .NET, CLR, etc.
3. ECMA Script, CSS, HTML, etc.

Unfortunately, it is inevitable to live in "clean concepts" without getting your hands dirty in a particular implementation. It does not mean that you need to know every single detail, but deep enough to understand all key concepts. Surprise, surprise! You will recognize most of the fundamentals mentioned above but with exact implementation details that are needed in real life. Your "whole picture" therefore will stay clear and bright.

Getting deeper means to selected stack means two directions:

• Low level technical details (like memory organizing, storing types, garbage collection, etc.)
• Architecture approaches (like ASP.NET MVC for .NET stack and PlayFramework for Java stack for Web and API applic)

Google Driven Development

At last, with solid fundamentals baggage, you can efficiently use all the power of Google to solve your real-world tasks. You might not remember a particular function name or the whole approach to solve a problem you faced on your daily work, but you are sure that there MUST BE a helpful function, component, approach, framework, etc. And you just need to ask Google appropriately. Or how to implement it using "copy-paste" approach. 

For example, if you have a list of values in most cases you transform them to another list (probably with some filtering), or you need to produce a single value based on all the elements. Back to your fundamentals, you KNOW that there are list comprehensions like filter(), map(), fold(), etc. And the only thing you need to search - how to use them in a particular language you work. Like list.map(...) in Scala is the same aslist.Select() in C#.

Another example is selecting a technology/framework to use for your project. When you read about a particular framework, you just match it with "checklist" of your understanding what this framework must contain to be selected and how it must be implemented in order to match your expectations. This is how CTOs work.