Torna al blog
Sviluppo software

Data Science with Python: the data structures and libraries I use every day in data analysis

Data Science doesn't start with Machine Learning models. It starts by understanding data, where it comes from and how to manipulate it. In this article I share the journey I take with Python: from lists and dictionaries to Pandas, NumPy and Natural Language Processing with spaCy.

Data Science starts with data, not models

There's a common misconception when starting a Data Science path: people think the core of the work is Machine Learning — models, algorithms, neural networks. In reality, 70-80% of a data scientist's time is spent on data: collecting, cleaning, transforming, understanding its structures. If you don't have a solid understanding of how data is organized and how Python manipulates it, the models you build on top will be fragile. Dirty data produces dirty models — the famous "garbage in, garbage out."

Types of data in the real world

Structured data — organized in tables with rows and columns: relational databases, Excel sheets, CSV files. Easiest to analyze because they have an explicit schema. Semi-structured data — have structure but not rigid: JSON, XML, HTML. Require more preprocessing work before analysis. Unstructured data — no predefined schema: texts, images, audio, video. The majority of real-world data, requiring specific techniques to transform into analyzable format.

Python data structures I use most

Lists with slice notation [start:stop:step] are foundational: dati[::2] for every other element, dati[::-1] to reverse, dati[-3:] for the last three. Dictionaries are key-value structures equivalent to JSON, with dict comprehensions for data transformation. Tuples for immutable data (coordinates, fixed configurations) — faster than lists and communicate intent. Sets for unique values, duplicate removal, and set operations: intersection (&), union (|), difference (-).

FIFO queues with deque: when performance matters

Python lists have O(n) performance for insertions/removals at the beginning. For large datasets with FIFO processing, this becomes a bottleneck. The solution is deque from collections: O(1) insertions and removals at both ends. The maxlen parameter is particularly useful for time windows — the deque automatically keeps only the last N elements.

NLP with spaCy: extracting information from text

spaCy is my go-to Python library for Natural Language Processing — fast, with pre-trained Italian models, and a clean API. Extracting nouns from Italian text to identify main topics is straightforward: load the Italian model, process the text, filter tokens by part-of-speech tag. From a list of nouns you can calculate frequencies to find dominant topics, filter stopwords, build a search index, or prepare data for a classification model.

NumPy: vectorized numerical computing

NumPy's strength is vectorization: instead of writing explicit Python loops over each element, you express operations on the entire array and NumPy executes them in optimized C. The speed difference is fundamental: an operation on a NumPy array of a million elements is orders of magnitude faster than the same calculation with a Python loop. NumPy is the foundation that sits under Pandas, scikit-learn, and virtually all of Python Data Science.

Pandas: the heart of structured data analysis

Series (one-dimensional array with labeled index, like an Excel column) and DataFrame (two-dimensional table with labeled rows and columns, like a full Excel sheet) are the fundamental structures. What would require JOIN, GROUP BY and complex subqueries in SQL can be expressed in three readable lines with Pandas. For large datasets, Pandas can delegate computation to Dask or Modin for parallelization.

The path toward Machine Learning

These skills — Python data structures, NumPy, Pandas, basic NLP — are not the destination. They are the starting point. Without these solid foundations, working with scikit-learn for Machine Learning or TensorFlow for neural networks becomes frustrating: you spend more time managing data than building models. The course builds these foundations progressively, with practical examples on real data, no prerequisites taken for granted.

Condividi