Learning Bits

Logging the progress.

nsprcomp is on CRAN

When we published our ICML paper on sparse and non-negative PCA back in 2008, I thought it might be worthwhile to provide Matlab code for the emPCA algorithm available as well. Since then, I’ve received several requests and questions about its usage. While the core functionality was there, the implementation lacked a friendly interface and some additional functionality such as easy random restarts.

After two recent inquiries about using constrained PCA for portfolio optimization and combustion modeling, I decided to fill the gaps in the implementation. Because I primarily use R now, this provided an opportunity to learn about package writing and documentation for a public audience, with the goal of submitting the result to CRAN.

The nsprcomp package provides two algorithms: nsprcomp() implements the emPCA algorithm as discussed in the paper, but with several deflation options for computing multiple components (motivated by this NIPS paper). nscumcomp() is a novel algorithm based on the same form of expectation-maximization, but it jointly computes all components such that the cumulative variance is maximized. One benefit of nscumcomp over nsprcomp is that the number of features can be specified as a total, instead of having to specify the cardinality of each principal axis individually. Setting the total is more natural in an exploratory data analysis, where the number of features that make up a component is not known in advance. A drawback of the joint optimization is that the M-step no longer has a closed form solution (that I know of), so the numerical optimization using L-BFGS iterations increases the computational load.

A small example from the domain of portfolio optimization can demonstrate the usefulness of both algorithms, but keep in mind that my knowledge about the subject essentially consist of reading Markowitz (1952) and a bunch of related papers. Let us assume that the variance in asset returns can be explained (approximately) using a linear combination of a small number of hidden factors, which of course suggests a principal component analysis. The drawback of classical PCA is that each factor consists of a linear combination of all assets with mixed signs. Enforcing non-negativity and sparsity of the loadings can result in a more meaningful analysis, as non-negative loadings correspond to long positions in the portfolio, and sparsity limits the number of positions in the portfolio.

We analyze NYSE daily returns for the year 2005, from data which is provided by the kind people at infochimps.com. After some pre-processing, the data matrix X contains 260 daily returns for 2365 stocks.

nsprcomp

Calling nsprcomp(X, ncomp=4, k=10, nneg=TRUE) returns four non-negative components with the top ten stocks each:

[[1]]
   weight symbol                                name            sector                                              industry
0.4434287    AKS        AK Steel Holding Corporation  Basic Industries                                        Steel/Iron Ore
0.3080191    ATI Allegheny Technologies Incorporated  Basic Industries                                        Steel/Iron Ore
0.3304431    CLF       Cliffs Natural Resources Inc.  Basic Industries                                       Precious Metals
0.3228908    CMC           Commercial Metals Company  Basic Industries                                        Steel/Iron Ore
0.2646535    CRS    Carpenter Technology Corporation  Basic Industries                                        Steel/Iron Ore
0.3076497    GPK   Graphic Packaging Holding Company Consumer Durables                                  Containers/Packaging
0.2925628    GTI          GrafTech International Ltd            Energy                       Industrial Machinery/Components
0.2947678     MT                       ArcelorMittal  Basic Industries                                        Steel/Iron Ore
0.2643825    NUE                   Nucor Corporation  Basic Industries                                        Steel/Iron Ore
0.2966061    USU                           USEC Inc.  Basic Industries Mining & Quarrying of Nonmetallic Minerals (No Fuels)

[[2]]
   weight symbol                                  name                sector                                              industry
0.2697512    ACO       Amcol International Corporation      Basic Industries Mining & Quarrying of Nonmetallic Minerals (No Fuels)
0.3291779    AIT Applied Industrial Technologies, Inc.     Consumer Durables                                Industrial Specialties
0.2878995    BKI            Buckeye Technologies, Inc.      Basic Industries                                                 Paper
0.3173853    CCC             Calgon Carbon Corporation      Basic Industries                                       Major Chemicals
0.3397139    CIA                        Citizens, Inc.               Finance                                        Life Insurance
0.3100144    CNC                   Centene Corporation           Health Care                                  Medical Specialities
0.2972055    ENS                               Enersys Consumer Non-Durables                          Telecommunications Equipment
0.3109916     NL                   NL Industries, Inc.      Basic Industries                                       Major Chemicals
0.4116398    PRM                                    NA                    NA                                                    NA
0.2631501    RKT                     Rock-Tenn Company     Consumer Durables                                  Containers/Packaging

[[3]]
   weight symbol                      name           sector     industry
0.3182184    BZH    Beazer Homes USA, Inc.    Capital Goods Homebuilding
0.3324353    DHI         D.R. Horton, Inc.    Capital Goods Homebuilding
0.3177888    HOV Hovnanian Enterprises Inc    Capital Goods Homebuilding
0.2986497    KBH                   KB Home    Capital Goods Homebuilding
0.2702869    LEN        Lennar Corporation Basic Industries Homebuilding
0.3632209    MTH      Meritage Corporation    Capital Goods Homebuilding
0.3124587    PHM          PulteGroup, Inc.    Capital Goods Homebuilding
0.2837270    RYL  Ryland Group, Inc. (The)    Capital Goods Homebuilding
0.3117444    SPF     Standard Pacific Corp    Capital Goods Homebuilding
0.3431356    TOL        Toll Brothers Inc.    Capital Goods Homebuilding

[[4]]
   weight symbol                             name sector                    industry
0.3072652    ARD                               NA     NA                          NA
0.2883218    CHK    Chesapeake Energy Corporation Energy        Oil & Gas Production
0.2663459    EAC                               NA     NA                          NA
0.3019455    FTO                               NA     NA                          NA
0.3332282   GMXR                               NA     NA                          NA
0.4095810    NGS Natural Gas Services Group, Inc. Energy Oilfield Services/Equipment
0.3135460     PQ            Petroquest Energy Inc Energy        Oil & Gas Production
0.3120241    SWN      Southwestern Energy Company Energy        Oil & Gas Production
0.3038500    TSO               Tesoro Corporation Energy    Integrated oil Companies
0.3058673    UPL            Ultra Petroleum Corp. Energy        Oil & Gas Production

The first component mostly consists of mining and steel companies. The second component is all over the place, but the third and fourth components are again very homogeneous (the NAs in the second and fourth component are due to unresolved stock symbols). The weights all have a similar magnitude and none of them is close to zero, which suggests that the per-component cardinality k could be increased to include more stocks in each factor (the NYSE energy sector for example contains over two hundred stocks). However, it would take a number of trial runs to optimize the cardinality of each component, such that all relevant and few spurious stocks are included in the analysis.

nscumcomp

Calling nscumcomp(X, ncomp=4, k=150, nneg=TRUE, gamma=1) instead lets the algorithm determine the cardinality of each component. Setting the orthogonality penalty is not sensitive in this example, and a value of gamma=1 results in essentially orthogonal components:

[[1]]
      weight symbol                                    name            sector                                              industry
0.0532875561    ACI                         Arch Coal, Inc.            Energy                                           Coal Mining
0.0694850581    ALY                                      NA                NA                                                    NA
0.0284222815    ANR           Alpha Natural Resources, inc.            Energy                                           Coal Mining
0.0836213427    APA                      Apache Corporation            Energy                                  Oil & Gas Production
0.0481126384    APC          Anadarko Petroleum Corporation            Energy                                  Oil & Gas Production
0.1178876782    ARD                                      NA                NA                                                    NA
0.1393779752    ATW                   Atwood Oceanics, Inc.            Energy                                  Oil & Gas Production
0.0190883361    BJS                                      NA                NA                                                    NA
0.0454705431    BPT            BP Prudhoe Bay Royalty Trust            Energy                              Integrated oil Companies
0.1322468235    BRY                 Berry Petroleum Company            Energy                                  Oil & Gas Production
0.1083120714    BTU              Peabody Energy Corporation            Energy                                           Coal Mining
0.1932201309    CHK           Chesapeake Energy Corporation            Energy                                  Oil & Gas Production
0.1346712775    CNQ      Canadian Natural Resources Limited            Energy                                  Oil & Gas Production
0.0708790208    CNX                      CONSOL Energy Inc.            Energy                                           Coal Mining
0.1469680348    COG             Cabot Oil & Gas Corporation            Energy                                  Oil & Gas Production
0.0398795193    COP                          ConocoPhillips            Energy                              Integrated oil Companies
0.0473333886    CPE                Callon Petroleum Company            Energy                                  Oil & Gas Production
0.0977452639    CRK                Comstock Resources, Inc.            Energy                                  Oil & Gas Production
0.0154340470    CRR                    Carbo Ceramics, Inc.     Capital Goods                       Industrial Machinery/Components
0.1081947055    DNR                  Denbury Resources Inc.            Energy                                  Oil & Gas Production
0.1181309078     DO         Diamond Offshore Drilling, Inc.            Energy                                  Oil & Gas Production
0.1192592682    DRQ                         Dril-Quip, Inc.            Energy                                    Metal Fabrications
0.1043657129    DVN                Devon Energy Corporation            Energy                                  Oil & Gas Production
0.1597735987    EAC                                      NA                NA                                                    NA
0.1396755779    ECA                      Encana Corporation            Energy                                  Oil & Gas Production
0.1329824122    EOG                     EOG Resources, Inc.            Energy                                  Oil & Gas Production
0.0938604480    ESV                               ENSCO plc            Energy                                  Oil & Gas Production
0.0249781513    FRO                          Frontline Ltd.    Transportation                                 Marine Transportation
0.0757529529    FST                  Forest Oil Corporation            Energy                                  Oil & Gas Production
0.1836804006    FTO                                      NA                NA                                                    NA
0.1837984763    GDP          Goodrich Petroleum Corporation            Energy                                  Oil & Gas Production
0.0671665209    GLF                 GulfMark Offshore, Inc.            Energy                                    Metal Fabrications
0.1445269400   GMXR                                      NA                NA                                                    NA
0.0837276244    HAL                     Halliburton Company            Energy                           Oilfield Services/Equipment
0.0627795330    HES                        Hess Corporation            Energy                              Integrated oil Companies
0.0036551657    HGT                   Hugoton Royalty Trust            Energy                                  Oil & Gas Production
0.1085047020    HLX      Helix Energy Solutions Group, Inc.            Energy                           Oilfield Services/Equipment
0.1106579862    HOC                                      NA                NA                                                    NA
0.0991632214    HOS              Hornbeck Offshore Services Consumer Services                                 Marine Transportation
0.0607473586     HP                 Helmerich & Payne, Inc.            Energy                                  Oil & Gas Production
0.0116382620    INT         World Fuel Services Corporation            Energy                                Oil Refining/Marketing
0.0645058955    IOC                    InterOil Corporation            Energy                                  Oil & Gas Production
0.0132567145    IYE                                      NA                NA                                                    NA
0.1853276399    KWK              Quicksilver Resources Inc.            Energy                                  Oil & Gas Production
0.0013747206    MDR           McDermott International, Inc.     Capital Goods                                    Metal Fabrications
0.0264661046    MEE                                      NA                NA                                                    NA
0.0729227215    MRO                Marathon Oil Corporation            Energy                                  Oil & Gas Production
0.0023811506    MUR                  Murphy Oil Corporation            Energy                              Integrated oil Companies
0.0412584481    NBL                       Noble Energy Inc.            Energy                                  Oil & Gas Production
0.0313167250    NBR                  Nabors Industries Ltd.            Energy                                  Oil & Gas Production
0.0563314301     NE                       Noble Corporation            Energy                                  Oil & Gas Production
0.1034268654    NFX            Newfield Exploration Company            Energy                                  Oil & Gas Production
0.1926936035    NGS        Natural Gas Services Group, Inc.            Energy                           Oilfield Services/Equipment
0.1008675352    NOV            National Oilwel Varcol, Inc.            Energy                                    Metal Fabrications
0.0991873145    NXY                                      NA                NA                                                    NA
0.0508974751    OII         Oceaneering International, Inc.            Energy                           Oilfield Services/Equipment
0.0993800241    OIS          Oil States International, Inc.            Energy                                    Metal Fabrications
0.0405026646    OXY        Occidental Petroleum Corporation            Energy                                  Oil & Gas Production
0.0495609266    PDE                                      NA                NA                                                    NA
0.1062543807    PKD                 Parker Drilling Company            Energy                                  Oil & Gas Production
0.1877221709     PQ                   Petroquest Energy Inc            Energy                                  Oil & Gas Production
0.0670664696    PVA               Penn Virginia Corporation            Energy                                  Oil & Gas Production
0.0527486302    PXD       Pioneer Natural Resources Company            Energy                                  Oil & Gas Production
0.1055507210    PXP Plains Exploration & Production Company            Energy                                  Oil & Gas Production
0.1053442278    RDC                     Rowan Companies plc            Energy                                  Oil & Gas Production
0.0690536207    RES                               RPC, Inc.            Energy                           Oilfield Services/Equipment
0.1143500050    RIG                         Transocean Ltd.            Energy                                  Oil & Gas Production
0.1532407298    RRC             Range Resources Corporation            Energy                                  Oil & Gas Production
0.1101877769    SFY                    Swift Energy Company            Energy                                  Oil & Gas Production
0.0715891522    SGY                Stone Energy Corporation            Energy                                  Oil & Gas Production
0.0223880237    SII                                      NA                NA                                                    NA
0.0534550114    SJT            San Juan Basin Royalty Trust            Energy                                  Oil & Gas Production
0.1063493615     SM                       SM Energy Company            Energy                                  Oil & Gas Production
0.1062401621    SPN          Superior Energy Services, Inc.            Energy                           Oilfield Services/Equipment
0.1152037963     SU                     Suncor Energy  Inc.            Energy                              Integrated oil Companies
0.1026210927    SUN                                      NA                NA                                                    NA
0.2121555712    SWN             Southwestern Energy Company            Energy                                  Oil & Gas Production
0.0702312918    TDW                          Tidewater Inc. Consumer Services                                 Marine Transportation
0.1125170113    TLM                    Talisman Energy Inc.            Energy                                  Oil & Gas Production
0.0005959932    TMR                                      NA                NA                                                    NA
0.0131588866     TS                            Tenaris S.A.  Basic Industries                                        Steel/Iron Ore
0.2152177841    TSO                      Tesoro Corporation            Energy                              Integrated oil Companies
0.0679062447    TTI                Tetra Technologies, Inc.            Energy                                  Oil & Gas Production
0.1174076026    UNT                        Unit Corporation            Energy                                  Oil & Gas Production
0.2088031326    UPL                   Ultra Petroleum Corp.            Energy                                  Oil & Gas Production
0.0273802381    USU                               USEC Inc.  Basic Industries Mining & Quarrying of Nonmetallic Minerals (No Fuels)
0.1610340003    VLO               Valero Energy Corporation            Energy                              Integrated oil Companies
0.0068010930    WFT          Weatherford International, Ltd            Energy                                  Oil & Gas Production
0.0952688259    WLL           Whiting Petroleum Corporation            Energy                                  Oil & Gas Production
0.0668893737    WMB          Williams Companies, Inc. (The)  Public Utilities                              Natural Gas Distribution
0.0138315166    XEC                       Cimarex Energy Co            Energy                                  Oil & Gas Production
0.1427017005    XTO                                      NA                NA                                                    NA
      weight symbol                                    name            sector                                              industry

[[2]]
    weight symbol                                 name           sector                                              industry
0.12797710    ABX             Barrick Gold Corporation Basic Industries                                       Precious Metals
0.17644436    AEM           Agnico Eagle Mines Limited Basic Industries                                       Precious Metals
0.11593957    ASA ASA Gold and Precious Metals Limited              n/a                                                   n/a
0.15675978     AU            AngloGold Ashanti Limited Basic Industries                                       Precious Metals
0.24193015    AUY                     Yamana Gold Inc. Basic Industries                                       Precious Metals
0.17403800    BVN     Buenaventura Mining Company Inc. Basic Industries                                       Precious Metals
0.01358500    CCJ                   Cameco Corporation Basic Industries                                       Precious Metals
0.40112842    CDE                   Coeur Mining, Inc. Basic Industries                                       Precious Metals
0.32590838    EGO            Eldorado Gold Corporation Basic Industries                                       Precious Metals
0.07069435    FCX Freeport-McMoran Copper & Gold, Inc. Basic Industries                                       Precious Metals
0.16373700    FDG                                   NA               NA                                                    NA
0.20941091    GFI                  Gold Fields Limited Basic Industries                                       Precious Metals
0.20107216     GG                        Goldcorp Inc. Basic Industries                                       Precious Metals
0.10346601    GRS                                   NA               NA                                                    NA
0.39352265     HL                 Hecla Mining Company Basic Industries Mining & Quarrying of Nonmetallic Minerals (No Fuels)
0.25090552    HMY  Harmony Gold Mining Company Limited Basic Industries                                       Precious Metals
0.16994222    IAG                  Iamgold Corporation Basic Industries                                       Precious Metals
0.01370509    IVN                                   NA               NA                                                    NA
0.25820797    KGC             Kinross Gold Corporation Basic Industries                                       Precious Metals
0.15124020    NEM           Newmont Mining Corporation Basic Industries                                       Precious Metals
0.08798759     RZ                                   NA               NA                                                    NA
0.03131700    SLW                  Silver Wheaton Corp Basic Industries                                       Precious Metals
0.29394621    SWC            Stillwater Mining Company Basic Industries                                       Precious Metals
    weight symbol                                 name           sector                                              industry

[[3]]
     weight symbol                                                           name                sector                            industry
0.096875336    ABV                      Companhia de Bebidas das Americas - AmBev Consumer Non-Durables Beverages (Production/Distribution)
0.326311357    BAK                                                   Braskem S.A.      Basic Industries                     Major Chemicals
0.250725988    BBD                                              Banco Bradesco Sa               Finance                         Major Banks
0.166407433   BRFS                                                       BRF S.A. Consumer Non-Durables                   Meat/Poultry/Fish
0.307049892    BTM                                                             NA                    NA                                  NA
0.089442235    CBD                           Companhia Brasileira de Distribuicao     Consumer Services                         Food Chains
0.321156820    CIG                                        Comp En De Mn Cemig ADS      Public Utilities         Electric Utilities: Central
0.116259858    CPL                                              CPFL Energia S.A.      Public Utilities         Electric Utilities: Central
0.144458384    CYD                             China Yuchai International Limited                Energy     Industrial Machinery/Components
0.309964150    ELP                        Companhia Paranaense de Energia (COPEL)      Public Utilities         Electric Utilities: Central
0.001868534    ERJ                      Embraer-Empresa Brasileira de Aeronautica         Capital Goods                           Aerospace
0.181906131    EWZ                                                             NA                    NA                                  NA
0.087424791    FBR                                           Fibria Celulose S.A.      Basic Industries                               Paper
0.274446999    GGB                                                    Gerdau S.A.         Capital Goods                      Steel/Iron Ore
0.050549732    GOL                            Gol Linhas Aereas Inteligentes S.A.        Transportation       Air Freight/Delivery Services
0.031581881    GPK                              Graphic Packaging Holding Company     Consumer Durables                Containers/Packaging
0.002041877    GTI                                     GrafTech International Ltd                Energy     Industrial Machinery/Components
0.035948636    ILF                                                             NA                    NA                                  NA
0.230162376   ITUB                                 Itau Unibanco Banco Holding SA               Finance                         Major Banks
0.166407433    PDA                                                             NA                    NA                                  NA
0.177097305    SBS Companhia de saneamento Basico Do Estado De Sao Paulo - Sabesp      Public Utilities                        Water Supply
0.026647018    TAR                                                             NA                    NA                                  NA
0.199002737    TBH                                                             NA                    NA                                  NA
0.211402025    TNE                                                             NA                    NA                                  NA
0.104866320    TSP                                                             NA                    NA                                  NA
0.239683225    TSU                                         TIM Participacoes S.A.      Public Utilities        Telecommunications Equipment
0.094630001   VALE                                                      VALE S.A.      Basic Industries                     Precious Metals
0.249716544    VIV                                         Telefonica Brasil S.A.      Public Utilities        Telecommunications Equipment
     weight symbol                                                           name                sector                            industry

[[4]]
     weight symbol                                  name                sector               industry
0.001306744    AIT Applied Industrial Technologies, Inc.     Consumer Durables Industrial Specialties
0.003829347    ALY                                    NA                    NA                     NA
0.023222920    ELN                 Elan Corporation, plc     Consumer Durables  Major Pharmaceuticals
0.151106602    KKD          Krispy Kreme Doughnuts, Inc. Consumer Non-Durables            Food Chains
0.000329151    MAG                                    NA                    NA                     NA
0.988199944    SGU               Star Gas Partners, L.P.     Consumer Services Other Specialty Stores
0.008478258    TGI                   Triumph Group, Inc.         Capital Goods              Aerospace

The cardinality of each component varies greatly in this analysis: there are 92 stocks in the first component which consists mostly of oil and gas production and some coal companies, and there are 23 stocks in the second component which is about gold and other precious metals (according to this page, the gold craze was already in effect back in 2005). The structure of the third component is curious: the stocks come from various sectors, but the common factor seems to be that most of them are Brazilian. The fourth is a junk component that essentially consists of Star Gas Partners and some Krispy Kreme — maybe the Star Gas employees like doughnuts?

Different random seeds reveal different sets of components with similar cumulative variances (including the mining and homebuilding components from the first analysis), but I stop here and leave further analysis to people who better know what they are doing.

References

H. Markowitz (1952). Portfolio Selection. The Journal of Finance, Vol. 7, No. 1., pp. 77-91.

Post a comment

All comments are held for moderation. Basic HTML formatting is accepted.

Name: (required)
Website: