Data Scientist interview questions and answers

A Python loop, and df.apply with axis=1, call Python code once per row, which is slow for millions of rows. Vectorised operations work on whole columns at once in optimised C code over NumPy arrays, often tens to hundreds of times faster.

So instead of applying a function that multiplies price and quantity per row, write df['revenue'] = df['price'] * df['qty']. For conditions, use np.where(df['amount'] > 1000, 'high', 'low'), or np.select for several conditions. The .str and .dt accessors handle text and dates column-wise too. apply is still fine for small data or logic that truly can't be vectorised, but say you'd measure before and after.

agg reduces each group to one row: df.groupby('customer_id')['amount'].agg('mean') gives one average per customer. transform returns a result the same length as the original DataFrame, with each row getting its group's value: df['cust_avg'] = df.groupby('customer_id')['amount'].transform('mean') puts each customer's average on every one of their orders.

Use transform when building features that compare a row with its group, such as an order's amount divided by the customer's average, or a rank or share within a group. It also avoids the extra step of aggregating and merging the result back. In time-based problems, make sure the group statistic uses only past data, or the feature leaks the future.

Long to wide is a pivot: df.pivot_table(index='customer_id', columns='month', values='spend', aggfunc='sum', fill_value=0). pivot_table also handles duplicate customer and month pairs by aggregating, where pivot would raise an error. Then flatten or rename the columns, for example to spend_2026_01, so they're valid feature names.

Wide to long is melt: df.melt(id_vars='customer_id', var_name='month', value_name='spend'). Long format suits groupby and plotting; wide suits most scikit-learn models. Think about whether 0 is right for missing months: a month before the customer joined isn't the same as a month with no spending, so you may want a separate tenure feature.

It warns that you may be changing a copy instead of the original DataFrame, so your change might be silently lost. It usually comes from chained assignment, like df[df['age'] > 30]['segment'] = 'A': the first part may return a copy, and the assignment changes only that copy.

The fix is to select and assign in one step with loc: df.loc[df['age'] > 30, 'segment'] = 'A'. If you deliberately want a separate subset to modify, make it explicit: subset = df[df['age'] > 30].copy(). From pandas 3.0, copy-on-write is the default, so chained assignment never changes the original and this warning is gone, which makes the loc habit even more important.

A p-value is the probability of getting a result at least as extreme as the one observed, assuming the null hypothesis is true. A p-value of 0.03 means that if there were really no difference, results this extreme or more would occur only about 3% of the time. With a significance level of 0.05, that's low enough to reject the null hypothesis.

What it doesn't mean: it isn't the probability that the null hypothesis is true, or a 97% chance the effect is real, and it says nothing about how big or useful the effect is. Interviewers listen for that distinction, so state it, and report the effect size with a confidence interval alongside it.

A Type I error is a false positive: rejecting the null hypothesis when it's actually true, such as concluding a new feature raised sales when it didn't. Its probability is the significance level, alpha, often set at 0.05. A Type II error is a false negative: failing to detect a real effect. Its probability is beta, and power is 1 minus beta, the chance of detecting an effect of a given size if it exists; 80% is a common target.

For a fixed sample size, lowering alpha raises beta. To reduce both, you need more data. In medical screening a false negative is usually costlier, while in spam filtering a false positive is, so the right balance depends on the business.

If you repeated the same study many times and built an interval the same way each time, about 95% of those intervals would contain the true value. So a 95% confidence interval of ₹420 to ₹480 for average order value means we're fairly confident the true average lies in that range, given how the interval was built.

Strictly, it doesn't mean there's a 95% probability that the true value is inside this particular interval; in the frequentist view, the true value is fixed and the interval either contains it or doesn't. A narrower interval means a more precise estimate, and it narrows as the sample grows. For a difference between two groups, an interval that excludes zero matches a significant result at the 5% level.

Use a t-test to compare means of a numeric variable. An independent two-sample t-test compares two groups, such as the average order value of users who saw a new layout versus those who didn't. Welch's version doesn't assume equal variances and is a safe default. A paired t-test is for before-and-after on the same units.

Use a chi-square test of independence for categorical counts: whether conversion (yes or no) depends on the version shown, using the contingency table of counts. It needs reasonably large expected counts in each cell; Fisher's exact test suits small ones. For heavily skewed numeric data with small samples, a non-parametric test such as Mann-Whitney U is an option. Mention checking the assumptions first.

First define the primary metric (say, checkout conversion) and guardrail metrics that mustn't get worse, like average order value and payment failures. Choose the randomisation unit, usually the user rather than the session, so each person sees one version consistently. Decide the minimum effect worth detecting, then calculate the sample size from it with alpha of 0.05 and 80% power, and from that the duration, in whole weeks so weekday patterns balance.

While it runs, check that the split is really 50/50 (a sample ratio mismatch points to a bug) and don't stop early when the p-value first dips below 0.05; peeking inflates false positives. Afterwards, report the effect with a confidence interval, and watch for novelty effects.

A model is always better on the data it learnt from, so we keep a test set it has never seen to estimate performance on new data. Typically: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42).

stratify=y keeps the class proportions the same in both sets, which matters for imbalanced data: with 5% fraud, a random split could give a test set with far fewer fraud cases. random_state fixes the shuffle so the split is reproducible. Use the test set only once, at the end; tune with a validation set or cross-validation. For time-based data, split by date instead of randomly.

fit learns parameters from data: a StandardScaler learns each column's mean and standard deviation, and a model learns its weights. transform applies learnt parameters to data. fit_transform does both in one step on the same data. predict uses a fitted model to produce outputs for new inputs.

So you call scaler.fit_transform(X_train) and then only scaler.transform(X_test). Fitting the scaler on the test set, or on all the data before splitting, lets information from the test data into training. That's data leakage: the evaluation looks better than it will be in production, where future data is never available when you fit. A Pipeline makes this mistake hard to make.

A ColumnTransformer applies different preprocessing to different columns, for example scaling numeric columns and one-hot encoding categorical ones, and a Pipeline chains that with the model into one object. That has three big benefits.

First, no leakage during cross-validation: the preprocessing is refitted inside each fold on that fold's training part only. Second, tuning: GridSearchCV can tune preprocessing and model settings together, using names like model__max_depth. Third, deployment: you save one fitted pipeline with joblib and call predict on raw input, so production applies exactly the same steps as training. Separate preprocessing code is the most common cause of a model behaving differently in production. Set handle_unknown='ignore' on the encoder for unseen categories.

Both try combinations of hyperparameters and score each with cross-validation: the training data is split into k folds, the model trains on k minus 1 and is scored on the remaining fold, k times, and the scores are averaged. GridSearchCV tries every combination in the grid, which becomes slow as the grid grows. RandomizedSearchCV samples a fixed number of combinations (n_iter) from ranges or distributions, and usually finds a good result much faster when there are many parameters.

Use StratifiedKFold for imbalanced classes and TimeSeriesSplit for time data. By default the best combination is refitted on the full training set. Then evaluate once on the untouched test set; reporting the cross-validation score as final is optimistic.

Linear regression predicts a continuous number, such as a house price, as a weighted sum of the features, and is usually fitted by minimising the squared errors. Logistic regression is for classification: it computes a weighted sum too, but passes it through the sigmoid function to get a probability between 0 and 1, such as the chance a customer churns. It's fitted by minimising log loss, which is maximum likelihood.

Its coefficients act on the log-odds: a coefficient of 0.7 multiplies the odds by about e^0.7, roughly 2. The default cut-off of 0.5 for predicting a class can and often should be changed to suit the business cost of each type of error.

At each node the tree tries possible splits, such as age below 35, and picks the one that makes the child nodes purest. For classification, purity is measured with Gini impurity (1 minus the sum of squared class proportions) or entropy, and the best split gives the largest drop in impurity. For regression, the split that most reduces the squared error is chosen. It's greedy: it picks the best split now without looking ahead.

Left unchecked, a tree keeps splitting until each leaf is nearly pure, memorising noise in the training data. Limit it with max_depth, min_samples_leaf or pruning, or use an ensemble such as a random forest.

A random forest uses bagging: it trains many deep trees independently, each on a bootstrap sample of rows and a random subset of features at each split, and averages their predictions. Averaging reduces variance, so it's robust, hard to overfit badly and works well with little tuning.

Gradient boosting builds shallow trees one after another, each fitted to the errors (the gradient of the loss) of the trees so far, and adds them with a small learning rate. That mainly reduces bias, and well-tuned boosting, as in XGBoost, LightGBM or CatBoost, is often the most accurate on tabular data. It needs more careful tuning and early stopping. Start with a random forest as a strong baseline, and use boosting when the extra accuracy matters.

That large gap is overfitting, or high variance: the model has learnt noise and details specific to the training data that don't carry over. In bias-variance terms, it's too complex for the amount of data. The opposite, poor scores on both sets, would be high bias: the model is too simple.

To fix it: simplify the model (limit tree depth, raise min_samples_leaf), add regularisation, use fewer or better features, get more training data, or use bagging. Check with cross-validation and a learning curve. Also rule out problems with the split itself: duplicates or leaking features can push training accuracy up, and a test set from a different period may simply be different.

Both add a penalty on the size of the model's coefficients to the loss, discouraging overly complex models. L1 (Lasso) adds the sum of the absolute values of the coefficients. It tends to push some coefficients to exactly zero, so it also selects features, useful when you suspect many are irrelevant. L2 (Ridge) adds the sum of squared coefficients. It shrinks all of them smoothly towards zero without removing any, and handles correlated features more stably. Elastic Net combines the two.

The penalty's strength is a hyperparameter to tune with cross-validation. Scale features first, or the penalty treats them unfairly. In scikit-learn's LogisticRegression, C is the inverse of the strength: smaller C means stronger regularisation.

For a binary classifier, the confusion matrix counts four outcomes: true positives (predicted positive, actually positive), false positives (predicted positive, actually negative), true negatives and false negatives. Precision = TP / (TP + FP): of everything the model flagged, how much was right. Recall = TP / (TP + FN): of all the actual positives, how many the model caught.

For a loan default model, high precision means few good customers are wrongly rejected; high recall means few defaulters slip through. Raising the decision threshold usually increases precision and lowers recall. Say which matters more for the business before choosing a threshold.

A model that predicts "not fraud" for every transaction is 99% accurate and catches no fraud at all, so accuracy hides exactly the failure we care about. With imbalanced classes, look at the minority class directly.

Use recall to see what share of fraud is caught, precision to see how many alerts are real (each false alert costs an analyst's time or annoys a genuine customer), and the F1 score, their harmonic mean, when you need one number. The precision-recall curve and its area show performance across thresholds. Best of all, choose the threshold by the business cost of a missed fraud versus a false alarm.

The ROC curve plots the true positive rate (recall) against the false positive rate at every threshold. The area under it, AUC, equals the probability that the model gives a randomly chosen positive a higher score than a randomly chosen negative: 0.5 is random, 1.0 is perfect. It measures how well the model ranks, independent of any threshold.

With heavy imbalance, ROC-AUC can look excellent because the false positive rate stays tiny when negatives are abundant, even if most alerts are false. The precision-recall curve focuses on the positive class and shows that problem clearly, so prefer PR-AUC for fraud or rare disease detection. Either way, still choose an operating threshold for deployment.

MAE is the average absolute error, in the data's units: "we're off by 120 units a day on average". It's easy to explain and less affected by a few large errors. RMSE squares errors before averaging, so it punishes large misses more; use it when big errors are especially costly, like stock-outs during a festival. R² is the share of variance explained compared with predicting the mean; it can even be negative on test data, and it doesn't tell the business how large the errors are.

MAPE, the percentage error, is popular with business teams but breaks down when actual values are zero or tiny. I'd report MAE or RMSE against a simple baseline, such as last year's same week.

One-hot encoding creates a 0/1 column per category. Use it for nominal categories with no order, like city or payment method, especially for linear models, which would otherwise treat numbers as having an order. Ordinal encoding maps categories to integers. It's right when there's a real order, like low, medium, high, and it's often fine for tree models, which can split on any number.

For high-cardinality columns, such as thousands of pin codes, one-hot creates too many columns; options are grouping rare categories, frequency encoding or target encoding. Target encoding must be computed out of fold, as scikit-learn's TargetEncoder does, or it leaks the target. Also plan for categories that appear only in production.

Scaling matters for models that use distances or gradient-based optimisation, or that penalise coefficient size: k-nearest neighbours, k-means, SVMs, PCA, linear and logistic regression with regularisation, and neural networks. Without it, a feature like annual income in lakhs dominates one like age just because of its range.

StandardScaler gives each feature mean 0 and standard deviation 1; MinMaxScaler maps to a 0 to 1 range but is sensitive to outliers; RobustScaler uses the median and IQR for data with outliers. Tree-based models, such as decision trees, random forests and gradient boosting, don't need scaling, because they split on thresholds. Fit the scaler on training data only.

A score that good should make you suspect data leakage: information in training that won't exist at prediction time. Common causes: a feature recorded after the outcome, such as "cancellation reason" or "last login date" measured after the churn date; aggregates computed over the whole history, including the future; preprocessing fitted on all data before splitting; and the same customer appearing in both training and validation.

Check feature importance, since a single dominant feature is a red flag. Rebuild features as of a cut-off date, split by time (train on earlier months, validate on later ones) and group by customer. Also compare production inputs with training data, since a genuine data shift could be the cause too.

Start with the right metric, such as recall, precision, PR-AUC or a cost-based measure, since accuracy is meaningless here. Then try class weights: most scikit-learn models accept class_weight='balanced', which makes errors on the rare class cost more, and boosting libraries have similar options. Tuning the decision threshold on validation data often helps more than anything else.

Resampling is another option: undersample the majority class or oversample the minority, for example with SMOTE from imbalanced-learn. Apply it only to the training folds, inside the pipeline, never before the split, or the validation scores become meaningless. Also check whether collecting more positive examples is possible. Keep the test set at the real class ratio so results reflect production.

It depends on volume, accuracy needs, cost and whether labelled data exists. An LLM with a good prompt can work with no labelled data and handle new topics, so it's quick to start, but each review costs tokens, it's slower, and results can vary. A trained classifier, such as logistic regression on text features or a small fine-tuned transformer, is cheap and fast to run at scale once you have labels.

A common practical approach combines them: use an LLM to label a sample, have people check it, train a small model on those labels for the full volume, and send uncertain cases to the LLM. Either way, measure accuracy on a human-labelled test set.

An LLM produces a probability for each possible next token, and temperature controls how it samples from them. Low temperature sharpens the distribution, so the most likely tokens are chosen and outputs become more consistent; high temperature flattens it, giving more varied and creative text, and more mistakes. Top-p is a related setting that samples only from the most likely tokens covering a set share of probability.

For extraction or classification, use a low temperature, such as 0, for consistency. Even then outputs aren't always identical between runs, so a pipeline should validate the output. Higher values suit brainstorming or marketing copy. Some newer models fix or ignore these settings, so check the provider's documentation.

Treat the LLM like any model and evaluate it. Have people label a random sample, a few hundred tickets stratified across categories, with clear written definitions, and measure agreement between two human labellers first, since that sets a realistic ceiling. Then compare the LLM's labels with the human labels: overall accuracy, per-category precision and recall, and a confusion matrix to see which categories it mixes up.

Improve the prompt with category definitions, examples of tricky cases and an "other or unclear" option, then measure again on the same sample. Ask for structured output and validate it. Keep checking a small sample regularly, because ticket types and the model itself change over time.

Follow the project's real order and keep the business problem in front. A good structure: problem, data, approach, evaluation, result, and what you'd do differently. For example: "I built a model to predict which customers of [dataset or company] would churn. The data had [size and source]. After cleaning it, I created features like [two examples], compared logistic regression with a gradient boosting model using cross-validation, and chose [model] for its [metric] of [value]. The main finding was [insight]. If I did it again, I'd [improvement, e.g. split by time to avoid leakage]."

Expect deep follow-up questions on every choice, so only mention what you can defend.

Acknowledge it honestly and show what makes your work different, with evidence. For example: "Many people have done the Titanic and house-price projects, so I worked on a problem I chose myself: [project, e.g. predicting delivery delays using public data I collected]. I had to clean messy data, deal with leakage, and explain the results to [someone, e.g. a local business owner]. I also [extra, e.g. wrote up my analysis on GitHub or a blog, deployed it as a simple app]."

Show depth in one project rather than listing many, and connect it to the company's own problems.

Choose a real failure with a clear lesson, and own your part. Use STAR: "Our demand forecast looked excellent offline (situation), but store managers stopped using it within a month (task: find out why). I spoke to them and found it missed local festivals and gave no reason for its numbers (action). We added a festival calendar as a feature, showed the main drivers next to each forecast, and let managers adjust it. Adoption recovered and errors fell by [amount] (result)."

End with the lesson, such as involving users early and checking the model in real use, not just offline metrics.

Don't promise a number you can't control, but don't just refuse either. Explain what you can commit to and reframe the goal around the business outcome. For example: "I'd say I can't guarantee 95% before seeing the data, but I can give an estimate within [timeframe] after a quick baseline. Then I'd ask what accuracy is needed for: if it's catching fraud, missing 5% might be fine while too many false alarms isn't. I'd agree a target in their terms, like rupees saved or hours saved, and report progress against it."

Interviewers want honesty about uncertainty plus a practical path forward.

Use plain language, the business's own terms, and one concrete example. For example: "The model gives each customer a churn risk score. For this customer, it's high mainly because their orders dropped from four a month to one and they raised two complaints recently. Customers with this pattern left in about [x] out of 10 cases last year."

Talk about what drives predictions in general (a feature importance chart or SHAP values, explained simply), how often it's right, and what action to take. Be clear that it's a probability, not a certainty, and never hide its limits.