devtools::install_github("jbrowell/bulktrends")User Guide: bulktrends
Example usage of the bulktrends package (version 0.1.2).
Set-up
bulktrends can be installed from it’s GitHub repository using
Once installed, load the package and set the path to where we are storing data.
library(bulktrends)
data_directory <- "path/to/my_data/"Parallel computation is supported in bulktrends using on the future framework. Its use is optional but recommended. To use parallel computation, simply set a “plan”.
future::plan("multicore")To view progress bars, ask progressr to handle them.
progressr::handlers(global = TRUE)Monthly UK Trade Info Data
Load UK Trade Info Bulk Data Files
This package has been developed to work with HMRC Monthly Import data. For more details, see the package README. We can use it to download monthly trade data from the UK Trade Info and load either a single file or multiple files or all of the files in a directory (and subdirectories). This can take a while for multiple years of data, so it is recommended to save and load this data as a .gz file.
download_uktradeinfo_bulk(
dest_dir = file.path(data_directory,"imports/"),
type = "imports",
overwrite = FALSE
)
imports <- read_uktradeinfo(path = file.path(data_directory,"imports/"))
fwrite(imports, file = file.path(data_directory,"imports.gz"))
# imports <- fread(file.path(data_directory,"imports.gz"))Alternatively, to update an existing dataset, simply download the latest data using download_uktradeinfo_bulk() and then update your dataset using update_uktradeinfo(). This function compares the existing dataset with the latest data downloaded and updates it by adding new rows and removing any rows that are no longer present in the latest data. Finally, save the updated dataset as a .gz file.
update_uktradeinfo(
existing = imports,
path = file.path(data_directory,"imports")
)
fwrite(imports, file = file.path(data_directory,"imports.gz"))head(imports) PERREF TYPE MONTHAC COMCODE SITC COD_SEQ COD_ALPHA PORT_SEQ PORT_CODE
<int> <int> <int> <char> <char> <int> <char> <int> <char>
1: 201601 2 202507 01------ 00--- 1 FR 0 ZZZ
2: 201601 2 202507 01------ 00--- 3 NL 0 ZZZ
3: 201601 2 202507 01------ 00--- 4 DE 0 ZZZ
4: 201601 2 202507 01------ 00--- 5 IT 0 ZZZ
5: 201601 2 202507 01------ 00--- 7 IE 0 ZZZ
6: 201601 2 202507 01------ 00--- 8 DK 0 ZZZ
COO_SEQ COO_ALPHA MODE_OF_TRANSPORT STAT_VALUE NET_MASS SUMM_UNIT
<int> <char> <int> <num> <num> <char>
1: NA NA 298531 2091 000000000022
2: NA NA 319270 20318 000000000000
3: NA NA 555 25 000000000000
4: NA NA 5033 258 000000000000
5: NA NA 1332312 406697 000000795057
6: NA NA 37035 7805 000000000289
SUPRESSION FLOW REC_TYPE DATE_START DATE_END
<int> <char> <int> <IDat> <IDat>
1: 0 imp 0 2016-01-01 2016-01-31
2: 0 imp 0 2016-01-01 2016-01-31
3: 0 imp 0 2016-01-01 2016-01-31
4: 0 imp 0 2016-01-01 2016-01-31
5: 0 imp 0 2016-01-01 2016-01-31
6: 0 imp 0 2016-01-01 2016-01-31Things to be aware of
Country of origin reporting changes from 2022 onward, individual EU countries are specified where previously they were not. There is therefore a large increase in volume (number or rows) of import data (see methodology and quality report for more information on the data).
Low value Trade EU: Imports via port code QVV only report chapter of
COMCODEand do not includeNET_MASS.
Request Data from UK Trade Data API
Download look-up tables, e.g. for commodity codes and ports, and save, or load if already available.
Commodity <- uktrades_request(endpoint = "Commodity")$valuehead(Commodity) CommodityId Cn8Code Hs2Code Hs4Code Hs6Code
1 0 00 <NA> <NA> <NA>
2 1 01 01 <NA> <NA>
3 2 02 02 <NA> <NA>
4 3 03 03 <NA> <NA>
5 4 04 04 <NA> <NA>
6 5 05 05 <NA> <NA>
Hs2Description
1 <NA>
2 Live animals
3 Meat and edible meat offal
4 Fish and crustaceans, molluscs and other aquatic invertebrates
5 Dairy produce; birds' eggs; natural honey; edible products of animal origin, not elsewhere specified or included
6 Products of animal origin not elsewhere specified or included
Hs4Description Hs6Description SitcCommodityCode Cn8LongDescription
1 <NA> <NA> 0 -
2 <NA> <NA> 0 -
3 <NA> <NA> 0 -
4 <NA> <NA> 0 -
5 <NA> <NA> 0 -
6 <NA> <NA> 0 -Quick Visualisation
The comcode_plot() function produces time plots of imports by various measures included in import data, including mass, value (£) and volume (number of imports).
Here are some examples:
comcode_plot(imports, "01", comcode_lookup=Commodity)Warning: Removed 60 rows containing missing values or values outside the scale range
(`geom_bar()`).
comcode_plot(imports, "0602", variable="STAT_VALUE", comcode_lookup=Commodity)
comcode_plot(imports, "01", variable="volume",comcode_lookup = Commodity)
- Hierarchical classification of a given commodity code
comcode_description("01012990", Commodity)
Hierarchical Description for Code: 01012990
---------------------------------------------
Chapter (Hs2): 01 — Live animals
Heading (Hs4): 0101 — Live horses, asses, mules and hinnies
Subheading (Hs6): 010129 — Live horses (excl. pure-bred for breeding)
Commodity code (Cn8): 01012990 — Live horses (excl. for slaughter, pure-bred for breeding)
---------------------------------------------comcode_description("08039010", Commodity)
Hierarchical Description for Code: 08039010
---------------------------------------------
Chapter (Hs2): 08 — Edible fruit and nuts; peel of citrus fruits or melons
Heading (Hs4): 0803 — Bananas, incl. plantains, fresh or dried
Subheading (Hs6): 080390 — Fresh or dried bananas (excl. plantains)
Commodity code (Cn8): 08039010 — Bananas, fresh (excl. plantains)
---------------------------------------------comcode_description("16041428", Commodity)
Hierarchical Description for Code: 16041428
---------------------------------------------
Chapter (Hs2): 16 — Preparations of meat, fish or crustaceans, molluscs or other aquatic invertebrates
Heading (Hs4): 1604 — Prepared or preserved fish; caviar and caviar substitutes prepared from fish eggs
Subheading (Hs6): 160414 — Prepared or preserved tunas, skipjack and Atlantic bonito, whole or in pieces (excl. minced)
Commodity code (Cn8): 16041428 — Prepared or preserved skipjack, whole or in pieces (excl. minced, fillets known as "loins" and such products in vegetable oil)
---------------------------------------------Filter by commodity groups
bulktrends includes useful groupings of commodity codes, such as plants and animals relevant for Sanitary and Phytosanitary Controls (SPS). An object called comcode_groups is included in bulktrends and loaded along with the package. Warning: comcode_groups does not include any suppressed codes, e.g. “01——”, take care when using.
head(imports[COMCODE %in% comcode_groups$animal_SPS]) PERREF TYPE MONTHAC COMCODE SITC COD_SEQ COD_ALPHA PORT_SEQ PORT_CODE
<int> <int> <int> <char> <char> <int> <char> <int> <char>
1: 201601 1 202507 01012100 00150 1 FR 0 ZZZ
2: 201601 1 202507 01012100 00150 5 IT 0 ZZZ
3: 201601 1 202507 01012100 00150 7 IE 0 ZZZ
4: 201601 1 202507 01022130 00111 7 IE 0 ZZZ
5: 201601 1 202507 01022921 00119 7 IE 0 ZZZ
6: 201601 1 202507 01022949 00119 4 DE 0 ZZZ
COO_SEQ COO_ALPHA MODE_OF_TRANSPORT STAT_VALUE NET_MASS SUMM_UNIT
<int> <char> <int> <num> <num> <char>
1: NA NA 502526 3282 000000000007
2: NA NA 57 2386 000000000005
3: NA NA 3813813 8337 000000000018
4: NA NA 1252 600 000000000001
5: NA NA 989810 99680 000000000917
6: NA NA 369744 69200 000000000346
SUPRESSION FLOW REC_TYPE DATE_START DATE_END
<int> <char> <int> <IDat> <IDat>
1: 0 imp 0 2016-01-01 2016-01-31
2: 0 imp 0 2016-01-01 2016-01-31
3: 0 imp 0 2016-01-01 2016-01-31
4: 0 imp 0 2016-01-01 2016-01-31
5: 0 imp 0 2016-01-01 2016-01-31
6: 0 imp 0 2016-01-01 2016-01-31Anomaly detection: individual
The anomaly detection workflow considers two types of anomalies; structural breaks and temporary outliers.
For a given commodity code and quantity, a series of helper functions have been created; (i) extract_ts to extract an aggregated time series by summing all commodities with the specified code by time period, (ii) select_best_model to automatically find an optimal regression model from a pool of candidate models of base models and models with structural breaks, and (iii) detect_outliers to specifically detect temporary outliers (“AO” additive outliers, “TC” temporary changes, “IO” innovative outliers) using an outlier detection algorithm.
In this workflow, structural break detection is performed within select_best_model() by using break_detection = TRUE. If any break is identified, the detected break information is stored in the break_entry output.
When return_tso = TRUE in detect_outliers, the full output object returned by tsoutliers::tso() is retained, allowing users to inspect the detected outliers, model diagnostics, fitted regressors, and temporary outlier effects in detail.
ts_data <- extract_ts(imports, "03", quantity="NET_MASS")
selected_model <- select_best_model(ts_data, response_col = "NET_MASS", break_detection = TRUE)
detect_anomaly <- detect_outliers(data = selected_model$data,
quantity= "NET_MASS",
scale_ts = FALSE,
return_tso = TRUE,
xreg = model.matrix(selected_model$formula, data = selected_model$data))
print(selected_model$break_entry) type ind time
<char> <num> <IDat>
1: SB 72 2021-12-01print(detect_anomaly)$data
DATE_START NET_MASS linear_trend day_of_year annual_sin annual_cos
<IDat> <num> <int> <int> <num> <num>
1: 2016-01-01 37197596 1 1 0.01721336 0.9998518
2: 2016-02-01 36082935 2 32 0.52341561 0.8520775
3: 2016-03-01 40798380 3 61 0.86745635 0.4975133
4: 2016-04-01 34380960 4 92 0.99991666 -0.0129103
5: 2016-05-01 40242961 5 122 0.86314213 -0.5049611
---
119: 2025-11-01 32853271 119 305 -0.85876396 0.5123714
120: 2025-12-01 36259848 120 335 -0.49377555 0.8695894
121: 2026-01-01 32453617 121 1 0.01721336 0.9998518
122: 2026-02-01 31971021 122 32 0.52341561 0.8520775
123: 2026-03-01 38849633 123 60 0.85876396 0.5123714
segments
<char>
1: segment1
2: segment1
3: segment1
4: segment1
5: segment1
---
119: segment2
120: segment2
121: segment2
122: segment2
123: segment2
$outliers
[1] type ind coefhat tstat
<0 rows> (or 0-length row.names)
$tso
Series:
Regression with ARIMA(0,0,0) errors
Coefficients:
(Intercept) annual_sin annual_cos segmentssegment2
37552454.3 -1321850.1 -743600.3 -3514007.9
s.e. 380865.4 538244.2 539455.1 592561.6
annual_sin:segmentssegment2 annual_cos:segmentssegment2
2087462.3 -60714.3
s.e. 841693.2 835145.0
sigma^2 = 1.099e+13: log likelihood = -2018.15
AIC=4050.3 AICc=4051.27 BIC=4069.98
No outliers were detected.plot.tsoutliers(detect_anomaly$tso)'x' does not contain outliers to displayNULLAnomaly detection: large-scale
This procedure can be performed for a list of commodity codes using the detect_anomalies() function. It returns (i) a table of all detected anomalies (breaks and outliers) across all codes (ii) a table for each code containing the time index, the original time series and the associated anomaly covariates.
hs2_codes <- unique(substr(imports$COMCODE, 1,2))
hs2_anomalies <- detect_anomalies(import_data = imports,
codes = hs2_codes[1:5],
quantity = "NET_MASS",
scale_ts = TRUE,
freq = "month",
verbose = FALSE,
tso_params = list(cval= 4,
types = c("AO", "TC", "IO")))
print(hs2_anomalies$outliers) type ind time coefhat tstat code
<fctr> <num> <IDat> <num> <num> <char>
1: SB 38 2019-02-01 NA NA 01
2: TC 39 2019-03-01 -4.619512 -4.800133 01
3: AO 44 2019-08-01 -3.101146 -4.337859 01
4: SB 57 2020-09-01 NA NA 01
5: AO 107 2024-11-01 3.393672 5.031156 01
6: SB 51 2020-03-01 NA NA 02
7: TC 61 2021-01-01 -2.642911 -5.310712 02
8: TC 37 2019-01-01 2.846913 4.768187 03
9: SB 72 2021-12-01 NA NA 03
10: SB 39 2019-03-01 NA NA 04
11: SB 79 2022-07-01 NA NA 04
12: SB 105 2024-09-01 NA NA 05
model_formula
<char>
1: NET_MASS ~ linear_trend + segments + linear_trend:segments
2: NET_MASS ~ linear_trend + segments + linear_trend:segments
3: NET_MASS ~ linear_trend + segments + linear_trend:segments
4: NET_MASS ~ linear_trend + segments + linear_trend:segments
5: NET_MASS ~ linear_trend + segments + linear_trend:segments
6: NET_MASS ~ linear_trend + segments + linear_trend:segments
7: NET_MASS ~ linear_trend + segments + linear_trend:segments
8: NET_MASS ~ annual_sin + annual_cos + segments + annual_sin:segments + annual_cos:segments
9: NET_MASS ~ annual_sin + annual_cos + segments + annual_sin:segments + annual_cos:segments
10: NET_MASS ~ linear_trend + annual_sin + annual_cos + segments + linear_trend:segments + annual_sin:segments + annual_cos:segments
11: NET_MASS ~ linear_trend + annual_sin + annual_cos + segments + linear_trend:segments + annual_sin:segments + annual_cos:segments
12: NET_MASS ~ linear_trend + segments + linear_trend:segments
anomaly_type
<char>
1: Break
2: Outlier
3: Outlier
4: Break
5: Outlier
6: Break
7: Outlier
8: Outlier
9: Break
10: Break
11: Break
12: BreakDaily IPAFFS Data
Load IPAFFS Data Files
The Food Standards Agency (FSA) makes some Import of Products, Animals, Food and Feed System (IPAFFS) data public which we can also analyse with bulktrends. The data files contain daily data of high-risk products of animal origin and high-risk food of non-animal origin imported into the United Kingdom via approved designated ports (see the FSA website for more information). We can load a single csv file or all csv files in a directory (and subdirectories) using read_ipaffs().
imports_ipaffs <- read_ipaffs(path = paste0(data_directory,"imports_ipaffs"))For consistency, some column names have been standardised. For example, (i) date columns include DATE_START and DATE_END, (ii) all variations of net weight columns across .csv files have been renamed as NET_MASS and (iii) all variations of commodity code columns are renamed as COMCODE.
head(imports_ipaffs[,c(1:4,9,10,55,56,43,17)]) YearOfDeclaration QuarterOfDeclaration MonthOfDeclaration DayOfDeclaration
<char> <char> <char> <char>
1: 2017 Q1 1 23
2: 2017 Q1 2 10
3: 2017 Q1 2 15
4: 2017 Q1 2 15
5: 2017 Q1 2 16
6: 2017 Q1 2 16
ImporterCountry CountryOfOrigin DATE_START DATE_END COMCODE NET_MASS
<char> <char> <Date> <Date> <char> <num>
1: GB IN 2017-01-23 2017-01-23 230910 0.63
2: GB CN 2017-02-10 2017-02-10 230910 10502.00
3: GB AU 2017-02-15 2017-02-15 02044250 12575.70
4: GB AU 2017-02-15 2017-02-15 02044310 12575.70
5: GB NZ 2017-02-16 2017-02-16 02069099 17979.20
6: GB NZ 2017-02-16 2017-02-16 02044310 17952.00Data Visualisation
The comcode_plot() function can be used to also visualise daily time series. For the IPAFFS data, we can currently view NET_MASS and volume as follows:
comcode_plot(imports_ipaffs[DATE_START>="2024-01-01"],
code = "02",
variable = "NET_MASS",
comcode_lookup=Commodity)
comcode_plot(imports_ipaffs[DATE_START>="2024-01-01"],
code = "02",
variable = "volume",
comcode_lookup=Commodity)
ts_data <- extract_ts(imports_ipaffs,
code = "15",
quantity = "NET_MASS",
date_col = "DATE_START",
fill_missing = 0,
freq = "day")
plot(ts_data)
Anomaly detection: individual
The anomaly detection procedure described using the monthly HMRC data can also be applied to the daily IPAFFS data. This involves using the same functions, such as, extract_ts to obtain the time series of a given commodity code, select_best_model to identify structural breaks and select an appropriate model from the same set of candidate models, and using detect_outliers to identify the various types of temporary outliers.
ts_data <- extract_ts(imports_ipaffs[DATE_START>="2023-01-01"],
code = "07",
quantity = "NET_MASS",
date_col = "DATE_START",
fill_missing=0)
selected_model <- select_best_model(ts_data, response_col = "NET_MASS", break_detection = TRUE)
detect_anomaly <- detect_outliers(data = selected_model$data,
quantity= "NET_MASS",
scale_ts = FALSE,
return_tso = TRUE,
xreg = model.matrix(selected_model$formula, data = selected_model$data))
print(selected_model$break_entry)NULLprint(detect_anomaly)$data
DATE_START NET_MASS linear_trend day_of_week day_of_year holiday
<Date> <num> <int> <char> <int> <char>
1: 2023-01-01 25696.35 1 Sunday 1 <NA>
2: 2023-01-02 45436.36 2 Monday 2 New Year’s Day
3: 2023-01-03 49642.30 3 Tuesday 3 2nd January
4: 2023-01-04 63523.20 4 Wednesday 4 <NA>
5: 2023-01-05 37401.26 5 Thursday 5 <NA>
---
727: 2024-12-27 46578.50 727 Friday 362 <NA>
728: 2024-12-28 55447.37 728 Saturday 363 <NA>
729: 2024-12-29 47393.65 729 Sunday 364 <NA>
730: 2024-12-30 48780.25 730 Monday 365 <NA>
731: 2024-12-31 62017.95 731 Tuesday 366 <NA>
is_holiday annual_sin annual_cos AO71
<lgcl> <num> <num> <num>
1: FALSE 1.721336e-02 0.9998518 0
2: TRUE 3.442161e-02 0.9994074 0
3: TRUE 5.161967e-02 0.9986668 0
4: FALSE 6.880243e-02 0.9976303 0
5: FALSE 8.596480e-02 0.9962982 0
---
727: FALSE -5.161967e-02 0.9986668 0
728: FALSE -3.442161e-02 0.9994074 0
729: FALSE -1.721336e-02 0.9998518 0
730: FALSE 6.432571e-16 1.0000000 0
731: FALSE 1.721336e-02 0.9998518 0
$outliers
type ind time coefhat tstat
1 AO 71 71 183010.7 7.216368
$tso
Series: structure(c(25696.35, 45436.36, 49642.3, 63523.2, 37401.26, 45769.25, 41323.95, 48148.85, 34936.25, 46531.35, 64706.05, 39381.3, 52004.45, 44836.2, 61661.6, 54181.45, 45164.2, 44864.01, 62731.74, 86894.4, 68946.25, 60407.5, 68212.95, 83357.15, 101685.35, 64559.56, 44128.83, 73412.8, 69323.25, 42146.83, 44171.43, 55044.17, 60701.45, 80365.47, 74399.8, 57929.7, 55174.76, 46678, 44860.05, 78596.2, 26084.73, 58424.45, 55166, 58195.8, 64598.4, 72590.6, 62994.05, 62291.81, 63673.35, 43040.25, 48438.01, 104145.05, 53576.93, 40798.7, 149822.6, 57954.4, 46194, 54314.9, 61549, 151171.25, 48447.81, 73034.66, 76001, 51217.2, 35813.56, 75806.9, 79622.25, 65146.75, 69121.27, 29083.77, 240846.3, 52235.38, 52541.1, 136714.6, 60664.03, 65863.27, 79547.76, 47641.2, 27732.04, 30182.05, 47889.65, 60016.3, 48876.57, 108707.25, 76047.4, 152518.62, 50232.75, 88689.5, 47842.4, 41469.15, 54846.9, 34477.15, 57133.24, 66700.1, 81623.8, 57827.3, 45823.9, 54794.3, 47737, 29986.8, 122680.3, 164947.25, 61224.53, 60983.87, 43346.4, 53578.35, 26904.2, 80684.1, 170718.6, 78935.7, 86008.7, 61996.72, 38682.69, 50891.3, 87445.7, 60322.72, 50314.38, 53463.52, 32544.25, 42196.3, 43468.2, 78489.25, 146079.34, 58885.65, 53995.01, 62357.38, 53791.88, 54197.19, 75202.75, 105906.6, 70467.32, 36797.52, 81471.75, 72370.95, 34429.71, 67924.5, 78083.4, 42837.3, 67438.59, 75438.7, 56088.5, 64556.45, 88660.95, 92659.95, 72896.2, 62260.34, 35068.35, 79862.6, 77291.18, 52228.55, 109351.95, 69736.5, 53283.15, 48873.6, 76127, 72218.73, 35439.35, 74311.8, 35426.03, 79418.35, 88467.5, 46851.4, 61418.23, 75052.55, 45423.57, 41074, 82669.42, 51882.9, 51436.1, 43226.9, 54748.5, 71688.93, 29719.25, 70716.75, 68113.5, 81258.9, 41483.48, 29132.7, 157601.1, 101589.5, 79778.68, 71636.95, 35455.25, 34591.2, 53264.6, 196803.28, 55698.05, 58246.69, 88120.25, 47516, 33736.85, 40045.6, 58158.2, 61992, 47050.82, 26094.25, 71872.86, 49103, 31248.4, 47803.38, 43879.4, 44189.62, 58404.05, 49099.07, 34609.33, 29873.8, 46190.28, 99051.64, 46918.5, 33962.8, 116654.56, 71857.66, 48686.9, 54403.71, 125594.06, 69486.62, 102789.3, 58151.58, 26433.4, 79856.6, 90425.88, 112086, 47148, 98497.92, 67612.36, 52979.4, 63481.2, 98633.5, 82963.32, 66705.64, 76880.7, 49030.57, 51219.64, 42503.6, 75672.3, 119010.9, 58586.28, 50065.8, 45464.9, 97791.22, 44449.15, 91180.1, 87227.26, 137867.96, 66706.4, 43713.1, 53468.9, 68375.4, 86040.24, 71651.9, 56647.04, 78749.62, 41459.2, 68498.82, 76801.2, 204424.62, 81923.8, 58237.4, 122713.04, 63541.9, 56100.54, 68576.9, 125864.8, 73937.46, 73608.44, 76703.72, 49571.98, 84187.22, 65419.88, 216540.86, 81928.35, 72714.72, 86600.34, 72132.4, 119402.07, 65752.4, 211947.38, 103719.17, 104433.37, 73793.45, 64809.15, 83646.34, 110730.75, 100698.34, 77171.6, 56466.2, 125370.21, 54886.7, 53068.58, 67826.45, 75649.97, 94819.55, 164177.64, 79503.16, 42907.59, 64506.14, 80735.7, 55826.87, 57633.4, 56389.3, 190879.4, 84939.12, 60974.03, 88259.7, 75467.91, 81001.5, 67386.54, 83578.4, 91346.46, 45709.16, 95384.55, 56323.1, 101676.05, 113983.09, 89386.7, 70138.15, 43109.67, 85986.2, 109197.9, 86702, 68643.8, 72292.59, 54853.1, 55954, 63425.25, 84982.5, 68271.6, 60845.15, 59076.62, 51883.43, 86961.87, 111493.52, 52548.09, 71459.25, 40141.34, 81826.07, 50440.85, 37908.97, 55327.7, 90736.8, 52425.75, 85268.74, 102447.25, 77874.05, 51065.27, 114792.5, 81988.05, 66674.7, 90883.32, 66059.4, 83508.5, 79812.02, 101842.6, 111415.5, 44208.65, 38664.85, 48608.2, 69211.62, 46842.91, 30001.05, 67338.75, 34765, 41120.59, 73717.6, 52789.6, 105830.12, 49669.9, 98773.55, 28481.9, 67052.29, 44384.4, 75647.9, 73938.5, 38010.55, 41001.51, 84497.62, 32681.99, 58595.95, 45378.42, 90222.3, 41913.15, 85444.88, 63854.02, 37769.74, 62362.6, 38372.8, 77677.9, 56232.4, 65030.9, 69067.19, 84991.7, 62856.55, 48526.68, 64451.35, 63225.7, 129042.5, 73746.43, 68817.24, 68630.9, 58137.44, 39249.7, 68340.75, 95784.03, 72007.9, 78389, 81453.6, 82202.51, 65251.65, 50561.2, 85069.7, 88895.4, 53118, 73140.8, 49408.72, 32543.4, 57959.75, 106164.81, 102989.6, 87199.09, 49315.05, 49098.43, 50891.95, 39812.25, 50548.08, 68267.45, 49540.9, 44317.9, 76462.04, 39049.54, 70729.35, 37563.15, 51148.75, 71743.55, 66567.2, 80310.69, 53009.53, 79520.14, 73416.32, 54266.38, 35926.03, 70160.1, 58600.03, 128145.5, 55229.7, 82314.28, 54512.68, 62401.5, 84208.39, 79483.44, 86251.32, 123591.8, 49076.77, 77957.91, 67097.58, 54478.4, 55734.08, 49027.05, 67344.7, 72771.01, 70284.7, 74181.22, 66586, 70284.64, 44065.8, 156861.5, 98068.61, 60176.85, 93932.58, 86385.75, 61900.78, 38635.4, 63098.2, 49136.81, 68032.85, 44260.39, 82129.4, 45057.85, 81690.75, 49998.17, 106470.45, 58196.25, 38614.2, 78086.65, 45800.3, 71865.7, 76323.1, 62641.51, 59751.95, 45233.58, 119399.56, 64884.38, 44443.65, 69620.95, 94000.11, 41636.74, 29775.2, 61972.25, 46520.26, 34253.55, 72868.9, 55077.91, 64202.85, 33372.66, 63792.6, 75732.52, 93879.25, 28058.3, 107157.15, 61587.1, 61995.23, 61272.35, 75812.22, 74483.4, 48421.05, 76437.21, 57153.45, 65052.21, 61212.45, 100807.6, 36693, 83828.41, 86845.9, 54309.45, 39941.27, 99220.2, 64957.02, 34411.3, 66352.8, 55009.44, 37361.35, 48462.11, 117597.1, 81516.02, 35005.35, 87329.55, 68940.39, 48879.53, 69094.52, 45152.48, 59393.64, 36422.2, 120735.45, 37058.55, 51654.65, 51120.97, 46700.7, 76781.47, 92068.3, 52644, 70826.65, 42753.05, 108627.15, 48595.8, 97847.4, 52479.45, 59810, 73607.24, 58902.35, 68418.8, 55845.5, 87363.82, 68761.2, 52119.2, 111417.16, 80008.35, 39688.74, 71038.2, 61206.32, 46243.4, 38380.4, 73166.54, 67267.64, 41740.74, 40157.8, 64405.28, 55889.7, 107099.5, 122407.5, 47651.8, 30056.72, 59543.84, 39978.96, 81885.04, 77410.38, 77856.93, 44059.2, 43308.44, 39266.34, 64071.58, 145224.9, 34243.42, 71235.04, 68702.72, 36384.68, 36608.3, 89619.56, 82797.4, 96273.42, 96498.34, 71246.7, 55437.28, 52361.12, 63952.54, 55004.4, 56334.3, 100788.62, 45715.9, 59195.76, 69159.64, 64377, 63272.75, 42124.3, 102232, 71029, 46138.92, 84088, 60393.51, 58631.85, 74334.85, 65786.89, 59261.2, 71602.34, 66342.55, 72189.31, 57612.45, 110610, 100473.35, 79133.35, 61994.34, 67088.05, 68186.39, 67167.1, 74060.55, 100427.04, 69296.85, 61974.96, 65748.35, 64206.23, 73336.3, 102670.45, 164153.27, 57838.45, 72265.25, 57757.95, 68212.77, 54275.95, 64095.7, 180847.3, 61808.7, 99535.1, 59387.35, 69202.59, 64457.75, 76314.9, 156749.26, 83253.55, 50968.19, 83898.65, 87642.3, 41705.95, 59812.95, 85418.52, 110524.1, 61315.48, 49944.9, 74781.95, 48996.65, 41828.3, 53684.87, 57452.6, 72234.64, 60864.75, 73460.07, 73140.75, 38944.1, 81072.91, 74899.75, 44652.45, 60533.45, 94459.19, 99913.54, 98591.95, 92237.4, 75790.7, 41595.7, 71930.75, 83685.33, 85796.65, 73470.85, 64015.67, 54927.35, 39446.8, 38405.65, 85317.94, 173949.58, 115859.75, 67538.73, 82836.54, 50179.33, 44021.8, 68356.1, 70517.8, 60236.45, 78038.91, 98532.22, 67434.71, 72082.26, 70599.51, 59211.4, 59001.85, 55371.38, 99688.05, 91175.35, 72259.97, 81357.21, 55084.05, 63517.15, 124340.23, 87431.2, 55990.54, 33573, 61859.84, 66049.15, 40786.15, 61077.13, 72043.15, 46578.5, 55447.37, 47393.65, 48780.25, 62017.95), tsp = c(1, 731, 1), class = "ts")
Regression with ARIMA(0,0,0) errors
Coefficients:
(Intercept) linear_trend annual_sin annual_cos day_of_weekMonday
62761.472 -0.3449 -5980.016 729.8628 -1175.461
s.e. 3031.247 4.7963 1432.079 1317.0962 3484.753
day_of_weekSaturday day_of_weekSunday day_of_weekThursday
5115.700 469.6484 4507.452
s.e. 3492.994 3493.0670 3492.963
day_of_weekTuesday day_of_weekWednesday AO71
4707.848 27078.259 183010.7
s.e. 3484.744 3493.043 25360.5
sigma^2 = 644336457: log likelihood = -8445.41
AIC=16914.81 AICc=16915.25 BIC=16969.95
Outliers:
type ind time coefhat tstat
1 AO 71 71 183011 7.216plot.tsoutliers(detect_anomaly$tso)
Anomaly detection: large-scale
Here is also an example of using the detect_anomalies() function to a selected group of commodity codes from the IPAFFS dataset for large-scale anomaly detection. Based on IPAFFS data availability, the function can skip time series that are too short or incomplete. A verbose option is also provided, when set to TRUE, progress messages are displayed to help track execution steps.
Similarly, the function returns (i) a table of all detected anomalies (breaks and temporary outliers) across all codes, and (ii) a table for each code containing the time index, the original time series and the associated anomaly covariates.
hs2_codes <- sort(unique(substr(imports_ipaffs$COMCODE, 1,2)))
hs2_anomalies_netmass <- detect_anomalies(import_data = imports_ipaffs[DATE_START>="2023-01-01"],
codes = hs2_codes[1:5],
quantity = "NET_MASS",
scale_ts = TRUE,
freq = "day",
verbose = FALSE,
tso_params = list(cval= 4,
types = c("AO", "TC", "IO")))
hs2_anomalies_volume <- detect_anomalies(import_data = imports_ipaffs,
codes = hs2_codes[1:5],
quantity = "volume",
scale_ts = TRUE,
freq = "day",
verbose = FALSE,
tso_params = list(cval= 4,
types = c("AO", "TC", "IO")))
head(hs2_anomalies_netmass$outliers)
head(hs2_anomalies_volume$outliers)Note: The above anomaly detection procedures for daily data work but can take a while to run so their output has been excluded.