Collectible plant price prediction
A regression pipeline over ~27k web listings: an honest baseline, compared models and a clear diagnosis of why R² is 0.23.
R² on test
0.232
Baseline (median): −0.007
MAE on test
$37.65
RMSE ≈ $81.68
Clean listings
26,581
Out of 27,097 scraped, P1–P99 clipping
Models compared
4 + baseline
Ridge, Random Forest, Gradient Boosting and tuned RF
The problem
Sellers of collectible plants (Monstera, Philodendron, Aglaonema and others) don't know what price to list a rare piece at. From ~27k listings scraped from marketplaces such as Etsy, eBay, Mercari and Amazon, I built a regression model that suggests a competitive price range.
The focus wasn't the algorithm but the whole process: cleaning noisy web data, building features from free text, comparing against an honest baseline and understanding why the model has the performance ceiling it has.
End-to-end pipeline
- 1
Cleaning
Parsing the price (text with symbol and commas) into a number, numeric conversion of badly typed columns and date parsing.
- 2
Outliers
Clipping to the 1st and 99th percentiles ($4 to $809.11) because scraping brings extreme errors that dominate the squared loss. 26,581 rows remain.
- 3
Log-scale target
I used log1p(price): prices are roughly log-normal, and training on raw price creates heteroscedasticity and lets expensive plants dominate training.
- 4
Feature engineering
Pot size recovered from the title with regex, keyword flags (rare, large, small), days since last update, title and description length, and platform grouped into the top 10 plus "other".
- 5
Preprocessing
ColumnTransformer with StandardScaler for numeric features and OneHotEncoder with handle_unknown='ignore' so production doesn't fail on a new platform.
- 6
70/15/15 split
Train fits parameters, validation compares models and tunes hyperparameters, and test is used exactly once at the end.
R² by model
Ridge, Random Forest and Gradient Boosting on validation; the Random Forest tuned with RandomizedSearchCV (20 iterations, cv=3) on test.
What signals the model found
Using permutation importance (preferred over the trees' feature_importances_, which overrates high-cardinality variables):
- The selling platform is the most important variable, and the same listing is worth a different amount depending on the marketplace.
- Description and title length are associated with higher prices, probably because professional sellers write more.
- Marking a plant as rare multiplies the recommended price (in an Etsy example with a 4-inch pot, from $27 to $64, ~2.4×), and the effect of size is larger for rare plants.
The diagnosis: why R² is 0.23
The residual analysis shows marked heteroscedasticity: the model is accurate at the low end and fails at the high end, with errors above $700 on $750–$800 plants that it predicts at $40–$60. The residual distribution has a much longer positive tail than negative one, meaning it underestimates expensive pieces far more than it overestimates cheap ones.
The diagonal edge of the residuals-vs-predicted plot is not a model flaw but an artifact of clipping the price at $809: the maximum possible residual is 809 minus the prediction, a line with slope −1.
The root cause is in the data: the domain's most predictive variable is missing, the exact species or variety (an Albo Monstera isn't worth what a regular one is). That signal lives in the title as free text that this pipeline doesn't vectorize.
What can be claimed with this model
- Yes: an approximate price range (a band around the prediction) and the direction and relative size of the effects of rarity, platform and size.
- Yes: ranking listings to spot which ones look over- or under-priced relative to the market.
- No: the exact price of rare or expensive pieces, differences between specific species, or values outside the training range.
Data extraction with LLMs
Part of the work was building the data that feeds these models.
- 1
Collection
Scraping with Beautiful Soup and Selenium from multiple web sources, with unstructured data of uneven quality.
- 2
Extraction with Gemini
I automated the extraction of product sheets (name, price, pot size) with the Gemini API over parsed HTML, independently of each site's structure.
- 3
Structured loading
Validated JSON output loaded into SQL (PostgreSQL) for analysis and modeling.
What I would do next
- Vectorize title and description (TF-IDF or embeddings) to capture the species, which is the biggest expected impact.
- Target-encode the species or variety as its own variable.
- Try modern boosting (LightGBM) and deploy the estimate as an API with drift monitoring.