WordPress.org

Make WordPress Core

Opened 5 years ago

Last modified 5 weeks ago

#14254 reopened defect (bug)

update_meta_cache fails; query too large?

Reported by: newpixel Owned by:
Milestone: Future Release Priority: normal
Severity: normal Version: 2.9.2
Component: Cache API Keywords: has-patch 2nd-opinion needs-codex
Focuses: Cc:

Description (last modified by nacin)

In the file meta.php, around line 183 in the 'update_meta_cache()' function, it tries to do a query but I noticed this can fail (ie. crash wordpress) if there are too many post id's in the query.

The function is being called from query_posts(), with 'posts_per_page' set to -1.

An example query that crashed it:

SELECT post_id, meta_key, meta_value FROM wp_postmeta WHERE post_id IN (1009,1006,989,933,902,860,859,858,857,793,838,837,836,827,825,310,780,777,776,775,774,773,770,763,760,759,758,757,728,756,755,754,753,752,751,750,748,746,
732,736,729,726,725,724,723,722,720,719,717,716,715,710,709,503,692,289,625,268,593,583,582,332,32,30,28,26,24,22)

Maybe there is some limit associated with queries of this type internal to wordpress, as this query works fine in phpMySql.

Apologies if this is already reported or irrelevant in 3.0!

Attachments (2)

14524.diff (766 bytes) - added by wonderboymusic 19 months ago.
14254-batch-ids.diff (4.0 KB) - added by jipmoors 5 weeks ago.
Batching meta ids when updating cache

Download all attachments as: .zip

Change History (17)

comment:1 @nacin5 years ago

  • Description modified (diff)
  • Keywords reporter-feedback added; update meta cache query posts removed

Can you explain what "crash WordPress" means? That query should have no issue.

comment:2 @nacin4 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to worksforme
  • Status changed from new to closed

comment:3 @mpvanwinkle772 years ago

  • Cc mpvanwinkle77 added
  • Component changed from General to Cache
  • Keywords dev-feedback added
  • Resolution worksforme deleted
  • Status changed from closed to reopened

I've run into this issue ... though to clarify it exhausts the available PHP memory which is set to 512M. The cause in my case is that I am importing 37,000+ posts and so when when update_meta_cache_runs after the import it first has the massive query SELECT ... IN(37000 IDS!!!) but then it tries to loop over the values returned and add them to the cache.

I'm not sure what the best solution would be. In my case it's probably to import fewer posts at once. But it is worth considering how to make the operation more scalable.

Cheers,

Mike

comment:4 @SergeyBiryukov2 years ago

  • Milestone set to Awaiting Review

comment:5 @wonderboymusic2 years ago

1) Are you using an object cache?

2) What's your server's PHP memory_limit?

comment:6 @rahulroom2 years ago

  • Cc rahulroom added

Hi,

I am facing same problem with update_meta_cache.
I have 4464 pages. memory_limit is 256M.

Wordpress Version is 3.5.1
Page: wp-admin/edit.php?post_type=page

This is how query looks for 4464 pages and takes too long to execute:

Query: SELECT post_id, meta_key, meta_value FROM wp_postmeta WHERE post_id IN (2804,2795,2807,2794,2801,2782,2802,2805,2806,2790,2779,2783,2803,2781,2785,2789,2798,2784,2786,2800,2793,2792,2787,2791,2780,2788,2799,2796,2797,2833,2824,2836,2823,2830,2811,2831,2834,2835,2819,2808,2812,2832,2810,2814,2818,2827,2813,2815,2829,2822,2821,2816,2820,2809,2817,2828,2825,2826,2862,2853,2865,2852,2859,2840,2860,2863,2864,2848,2837,2841,2861,2839,2843,2847,2856,2842,2844,2858,2851,2850,2845,2849,2838,2846,2857,2854,2855,2891,2882,2894,2881,2888,2869,2889,2892,2893,2877,2866,2870,2890,2868,2872,2876,2885,2871,2873,2887,2880,2879,2874,2878,2867,2875,2886,2883,2884,2920,2911,2923,2910,2917,2898,2918,2921,2922,2906,2895,2899,2919,2897,2901,2905,2949,2940,2952,2939,2946,2927,2947,2950,2951,2935,2924,2928,2948,2926,2930,2934,2943,2929,2931,2945,2938,2937,2932,2936,2925,2933,2944,2941,2942,2914,2900,2902,2916,2909,2908,2903,2907,2896,2904,2915,2912,2913,42067,2978,2969,2981,2968,2975,2956,2976,2979,2980,2964,2953,2957,2977,2955,2959,2963,2972,2958,2960,2974,2967,2966,2961,2965,2954,2962,2973,2970,2971,3007,2998,3010,2997,3004,2985,3005,3036,3027,3039,3026,3033,3014,3034,3037,3038,3022,3011,3015,3035,3013,3017,3021,3030,3016,3018,3032,3025,3024,3019,3023,3012,3020,3031,3028,3029,3008,3009,3065,3056,3068,3055,3062,3043,3063,3066,3067,3051,3040,3044,3064,3042,3046,3050,3059,3045,3047,3061,3054,3053,3048,3052,3041,3049,3060,3057,3058,2993,2982,2986,3006,2984,2988,2992,3094,3085,3097,3084,3091,3072,3092,3095,3096,3080,3069,3073,3093,3071,3075,3079,3088,3074,3076,3090,3083,3082,3077,3081,3070,3078,3089,3086,3087,3302,3311,3299,3312,3305,3101,3304,3301,3300,3316,3098,3323,3303,3100,3321,3317,3308,3322,3320,3306,3313,3314,3319,3315,3099,3318,3307,3310,3309,3001,2987,2989,3003,2996,2995,2990,2994,2983,2991,3002,2999,3000,3273,3282,3270,3283,3276,3295,3275,3272,3271,3287,3298,3294,3274,3296,3292,3288,3279,3293,3291,3277,3284,3285,3290,3286,3297,3289,3278,3281,3280,5251,5263,5248,5256,5264,5257,5276,5255,5250,5249,5252,5268,5279,5275,5254,5253,5277,5273,5269,5260,5274,5272,5258,5265,5267,5271,5266,5278,5270,5259,5262,5261,5219,5231,5216,5224,5232,5225,5244,5223,5218,5217,5220,5236,5247,5243,5222,5221,5245,5241,5237,5228,5242,5240,5226,5233,5234,5239,5235,5246,5238,5227,5230,5229,5187,5199,5184,5191,5200,5193,5212,5192,5186,5185,5188,5204,5215,5211,5190,5189,5213,5209,5205,5196,5210,5208,5194,5201,5202,5207,5203,5214,5206,5195,5198,5197,1174,1166,1177,1173,1175,1171,1167,1172,1170,1163,1164,1169,1165,1176,1168,1159,1151,1162,1158,1160,1156,1152,1157,1155,1148,1149,1154,1150,1161,1153,1144,1136,1147,1143,1145,1141,1137,1142,1140,1193,1134,1139,1135,1146,1138,1197,1205,1194,1198,1196,1200,1204,1199,1201,1208,1207,1202,1206,1195,1203,1212,1220,1209,1213,1211,1215,1219,1214,1216,1223,1222,1217,1221,1210,1218,1227,1235,1224,1228,1226,1230,1234,1229,1231,1238,1237,1232,1236,1225,1233,1332,1340,1329,1333,1331,1335,1339,1334,1336,1343,1342,1337,1341,1330,1338,5154,5165,5151,5157,5166,5159,5178,5156,5153,5152,5158,5170,5181,5177,5155,5179,5175,5171,5162,5176,5174,5160,5167,5168,5173,5169,5180,5172,5161,5164,5163,5123,5134,5120,5126,5135,5128,5147,5125,5122,5121,5127,5139,5150,5146,5124,5148,5144,5140,5131,5145,5143,5129,5136,5137,5142,5138,5149,5141,5130,5133,5132,5092,5103,5089,5095,5104,5097,5116,5094,5091,5090,5096,5108,5119,5115,5093,5117,5113,5109,5100,5114,5112,5098,5105,5106,5111,5107,5118,5110,5099,5102,5101,3244,3253,3241,3254,3247,3266,3246,3243,3242,3258,3269,3265,3245,3267,3263,3259,3250,3264,3262,3248,3255,3256,3261,3257,3268,3260,3249,3252,3251,5030,5041,5027,5033,5042,5035,5054,5032,5029,5028,5034,5046,5057,5053,5031,5055,5051,5047,5038,5052,5050,5036,5043,5044,5049,5045,5056,5048,5037,5040,5039,4999,5010,4996,5002,5011,5004,5023,5001,4998,4997,5003,5015,5026,5022,5000,5024,5020,5016,5007,5021,5019,5005,5012,5013,5018,5014,5025,5017,5006,5009,5008,4968,4979,4965,4971,4980,4973,4992,4970,4967,4966,4972,4984,4995,4991,4969,4993,4989,4985,4976,4990,4988,4974,4981,4982,4987,4983,4994,4986,4975,4978,4977,3215,3224,3212,3225,3218,3237,3217,3214,3213,3229,3240,3236,3216,3238,3234,3230,3221,3235,3233,3219,3226,3227,3232,3228,3239,3231,3220,3223,3222,5061,5072,5058,5064,5073,5066,5085,5063,5060,5059,5065,5077,5088,5084,5062,5086,5082,5078,5069,5083,5081,5067,5074,5075,5080,5076,5087,5079,5068,5071,5070,3186,3195,3183,3196,3189,3208,3188,3185,3184,3200,3211,5182,5183,3207,3187,3209,3205,3201,3192,3206,3204,3190,3197,3198,3203,3199,3210,3202,3191,3194,3193,5507,5519,5504,5512,5520,5513,1422,5532,5511,5506,5505,5508,1430,5524,1419,5534,1423,5531,5510,5509,1421,5533,1425,5529,1429,5525,3157,3166,3154,3167,3160,3179,3159,3156,3155,3171,3182,3178,3158,3180,3176,3172,3163,3177,3175,3161,3168,3169,3174,3170,3181,3173,3162,3165,3164,5516,1424,5530,1426,5528,5514,1433,5521,1432,5522,1427,5527,1431,5523,1420,5535,1428,5526,5515,5518,5517,1370,1460,1359,1449,1363,1453,4534,4545,4531,4536,4546,4539,4558,4537,4533,4532,4538,4550,4561,4557,4535,4559,4555,4551,4542,4556,4554,4540,4547,4548,4553,4549,4560,4552,4541,4544,4543,4907,4917,4903,4909,4918,4911,4930,4908,2776,3108,2773,3109,3102,3121,2778,2775,2774,3113,3124,3120,2777,3122,3118,3114,3105,3119,3117,3103,3110,3111,3116,3112,3123,3115,3104,3107,3106,4905,4904,4910,4922,4933,4929,4906,4931,4927,4923,2747,2756,2744,2757,2750,2769,2749,2746,2745,2761,2772,2768,2748,2770,2766,2762,2753,2767,2765,2751,2758,2759,2764,2760,2771,2763,2752,2755,2754,4914,4928,4926,4912,4919,4920,4925,4921,4932,4924,4913,4916,4915,1347,1467,1355,1475,1344,1464,1348,1468,1346,1466,1350,1470,1354,1474,2718,2727,2715,2728,2721,2740,2720,2717,2716,2732,2743,2739,2719,2741,2737,2733,2724,2738,2736,2722,2729,2730,2735,2731,2742,2734,2723,2726,2725,1349,1469,1351,1471,1358,1478,1357,1477,1352,1472,1356,1476,1345,1465,1353,1473,1361,1437,1445,1434,1438,1436,1440,1444,1439,1441,1448,1447,1442,1446,1435,1443,3128,3137,3125,3138,3131,3150,3130,3127,3126,3142,3153,3149,3129,3151,3147,3143,3134,3148,3146,3132,3139,3140,3145,3141,3152,3144,3133,3136,3135,4875,4886,4872,4878,4887,4880,4899,4877,4874,4873,4879,4891,4902,4898,4876,4900,4896,4892,4883,4897,4895,4881,4888,4889,4894,4890,4901,4893,4882,4885,4884,4844,4855,4841,4847,4856,4849,4868,4846,4843,4842,4848,4860,4871,4867,4845,4869,4865,4861,4852,4866,4864,4850,4857,4858,4863,4859,4870,4862,4851,4854,4853,348,4813,4824,4810,4816,4825,4818,4837,4815,4812,4811,4817,4829,4840,4836,4814,4838,4834,4830,4821,4835,4833,4819,4826,4827,4832,4828,4839,4831,4820,4823,4822,4658,4782,5475,4669,4793,5487,4655,4779,5473,4661,4785,5480,4670,4794,5488,4663,4787,5481,1287,4682,4806,5500,4660,4784,5479,4657,4781,5474,4656,4780,5472,1302,1310,1299,1303,1301,1305,1309,1304,1306,1313,1312,1307,1311,1300,1308,4662,4786,5477,1295,4674,4798,5492,1317,1325,1314,1318,1316,1320,1324,1319,1321,1328,1327,1322,1326,1315,1323,1284,4685,4809,5503,1288,4681,4805,5499,5478,4659,4783,5476,1286,4683,4807,5501,4720,4731,4717,4723,4732,4725,4744,4722,4719,4718,4724,4736,4747,4743,4721,4745,4741,4737,4728,4742,4740,4726,4733,4734,4739,4735,4746,4738,4727,4730,4729,4689,4700,4686,4692,4701,4694,4713,4691,4688,4687,4693,4705,4716,4712,4690,4714,4710,4706,4697,4711,4709,4695,4702,4703,4708,4704,4715,4707,4696,4699,4698,1290,4679,4803,5497,1294,4675,4799,5493,1482,1490,1479,1483,1481,1485,1489,1484,1486,1493,1492,1487,1491,1480,1488,4666,4790,5484,1289,4680,4804,5498,1291,4678,4802,5496,4664,4788,5482,1298,4671,4795,5489,1297,4672,4796,5490,1292,4677,4801,5495,1296,4673,4797,5491,1285,4684,4808,5502,1293,4676,4800,5494,4665,4789,5483,4668,4792,5485,4667,4791,5486,4751,5442,4762,5455,4748,5440,4754,5449,4763,5456,4756,5448,1377,4775,5468,4753,5447,4750,5443,4749,5441,2689,2698,2686,2699,2692,2711,2691,2688,2687,2703,2714,2710,2690,2712,2708,2704,2695,2709,2707,2693,2700,2701,2706,2702,2713,2705,2694,2697,2696,1497,1505,1494,1498,1496,1500,1504,1499,1501,1508,1507,1502,1506,1495,1503,4755,5444,1512,1520,1509,1513,1511,1515,1519,1514,1516,1523,1522,1517,1521,1510,1518,1385,4767,5460,2660,2669,2657,2670,2663,2682,2662,2659,2658,2674,2685,2681,2661,2683,2679,2675,2666,2680,2678,2664,2671,2672,2677,2673,2684,2676,2665,2668,2667,1374,4778,5471,1378,4774,5467,5446,4752,5445,1376,4776,5469,1380,4772,5465,1384,4768,5461,2631,2640,2628,2641,2634,2653,2633,2630,2629,2645,2655,2652,2632,2654,2651,2646,2637,2650,2648,2635,2642,2643,2649,2644,2656,2647,2636,2639,2638,4759,5452,1379,4773,5466,1381,4771,5464,4757,5450,1388,4764,5457,1387,4765,5458,1382,4770,5463,1386,4766,5459,1375,4777,5470,1383,4769,5462,4758,5451,4761,5454,4760,5453,1364,1454,1366,1456,4627,4638,4624,4630,4639,4632,4651,4629,4626,4625,4631,4643,4654,4650,4628,4652,4648,4644,4635,4649,4647,4633,4640,4641,4646,4642,4653,4645,4634,4637,4636,1373,1463,1372,1462,1367,1457,1371,1461,1360,1450,1368,1458,4596,4607,4593,4599,4608,4601,4620,4598,4595,4594,4600,4612,4623,4619,4597,4621,4617,4613,4604,4618,4616,4602,4609,4610,4615,4611,4622,4614,4603,4606,4605,4565,4576,4562,4568,4577,4570,4589,4567,4564,4563,4569,4581,4592,4588,4566,4590,4586,4582,4573,4587,4585,4571,4578,4579,4584,4580,4591,4583,4572,4575,4574,2601,2612,2598,2611,2604,2623,2605,2602,2600,2616,2627,2624,2603,2625,2620,2618,2609,2622,2621,2606,2613,2615,2619,2614,2626,2617,2607,2610,2608,4503,4514,4500,4506,4515,4508,4527,4505,4502,4501,4507,4519,4530,4526,4504,4528,4524,4520,4511,4525,4523,4509,4516,4517,4522,4518,4529,4521,4510,4513,4512,41,1527,1535,1524,1528,1526,1530,1534,1529,1531,1538,1537,1532,1536,1525,1533,1542,1550,1539,1543,1541,1545,1549,1544,1546,1553,1552,1547,1551,1540,1548,1557,1565,1554,1558,1556,1560,1564,1559,1561,1568,1567,1562,1566,1555,1563,1572,1580,1569,1573,1571,1575,1579,1574,1576,1583,1582,1577,1581,1570,1578,3339,3330,3342,3329,3336,2594,3337,3340,3341,3325,2599,2595,3338,2596,2593,3324,3333,2592,2591,3335,3328,3327,2590,3326,2597,2589,3334,3331,3332,3368,3359,3371,3358,3365,3346,3366,3369,3370,3354,3343,3347,3367,3345,3349,3353,3362,3348,3350,3364,3357,3356,3351,3355,3344,3352,3363,3360,3361,3397,3388,3400,3387,3394,3375,3395,3398,3399,3383,3372,3376,3396,3374,3378,3382,3391,3377,3379,3393,3386,3385,3380,3384,3373,3381,3392,3389,3390,3426,3417,3429,3416,3423,3404,3424,3427,3428,3412,3401,3405,3425,3403,3407,3411,3420,3406,3408,3422,3415,3414,3409,3413,3402,3410,3421,3418,3419,1587,1595,1584,1588,1586,1590,1594,1589,1591,1598,1597,1592,1596,1585,1593,3455,3446,3458,3445,3452,3433,3453,3456,3457,3441,3430,3434,3454,3432,3436,3440,3449,3435,3437,3451,3444,3443,3438,3442,3431,3439,3450,3447,3448,1602,1610,1599,1603,1601,1605,1609,3484,3475,3487,3474,3481,3462,3482,3485,3486,3470,3459,3463,3483,3461,3465,3469,3478,3464,3466,3480,3473,3472,3467,3471,3460,3468,3479,3476,3477,1604,1606,1613,1612,1607,1611,1600,1608,1617,1625,1614,1618,1616,1620,1624,1632,1640,1629,1633,1631,1635,1639,1634,1636,1643,1642,1637,1641,1630,1638,1647,1655,1644,1648,1646,1650,1654,1649,1651,1658,1657,1652,1656,1645,1653,1619,1621,1628,1627,1622,1626,1615,1623,426,3513,3504,3516,3503,3510,3491,3511,3514,3515,3499,3488,3492,3512,3490,3494,3498,3507,3493,3495,3509,3502,3501,3496,3500,3489,3497,3508,3505,3506,4472,4483,4469,4475,4484,4477,4496,4474,4471,4470,4476,4488,4499,4495,4473,4497,4493,4489,4480,4494,4492,4478,4485,4486,4491,4487,4498,4490,4479,4482,4481,4441,4452,4438,4444,4453,4446,4465,4443,4440,4439,4445,4457,4468,4464,4442,4466,4462,4458,4449,4463,4461,4447,4454,4455,4460,4456,4467,4459,4448,4451,4450,1677,1685,1674,1678,1676,1680,1684,1679,1681,1688,1687,1682,1686,1675,1683,4410,4420,4408,4413,4422,4414,4434,4411,4409,4407,4415,4427,4437,4432,4412,4435,4431,4426,4417,4433,4430,4416,4423,4424,4429,4425,4436,4428,4418,4421,4419,4379,4389,4376,4382,4391,4384,4402,4381,4378,4377,4383,4394,4406,4403,4380,4404,4401,4396,4388,4400,4399,4385,4393,4392,4397,4395,4405,4398,4386,4390,4387,1662,1692,1700,1689,1693,1691,1695,1699,1694,1696,1703,1702,1697,1701,1690,1698,1670,1659,1663,1661,1665,1669,5412,5422,5408,5415,5425,5418,1707,5436,5416,5410,5409,5411,1715,5428,1704,5439,1708,5434,5413,5414,1706,5437,1710,5433,1714,5429,5419,1709,5435,1711,5432,5417,1718,5424,1717,5426,1712,5431,1716,5427,1705,5438,1713,5430,5420,5423,5421,1664,1666,1673,1672,1667,1671,1660,1668,1722,1730,1719,1723,1721,1725,1729,1737,1745,1734,1738,1736,1740,1744,1739,1741,1748,1747,1742,1746,1735,1743,1752,1760,1749,1753,1751,1755,1759,1754,1756,1763,1762,1757,1761,1750,1758,1724,1726,1733,1732,1727,1731,1720,1728,1767,1775,1764,1768,1766,1770,1774,1769,1771,1778,1777,1772,1776,1765,1773,1782,1790,1779,1783,1781,1785,1789,1784,1786,1793,1792,1787,1791,1780,1788,1797,1805,1794,1798,1796,1800,1804,1799,1801,1808,1807,1802,1806,1795,1803,1812,1820,1809,1813,1811,1815,1819,1814,1816,1823,1822,1817,1821,1810,1818,1827,1835,1824,1828,1826,1830,1834,1829,1831,1838,1837,1832,1836,1825,1833,5379,5391,5376,5384,5392,5385,1842,5404,5383,5378,5377,5381,1850,5396,1839,5407,39715,1843,5403,5382,5380,1841,5405,1845,5401,1849,5397,5388,1844,5402,1846,5400,5386,1853,5393,1852,5394,1847,5399,1851,5395,1840,5406,1848,5398,5387,5390,5389,5347,1857,1865,1854,1858,1856,1860,1864,1859,1861,1868,1867,1862,1866,1855,1863,5359,5344,5353,5361,5352,1392,5372,5351,1872,1880,1869,1873,1871,1875,1879,1874,1876,1883,1882,1877,1881,1870,1878,3542,3533,3545,3532,3539,3520,3540,3543,3544,3528,3517,3521,3541,3519,3523,3527,3536,3522,3524,3538,3531,3530,3525,3529,3518,3526,3537,3534,3535,5346,5345,3571,3562,3574,3561,3568,3549,3569,3572,3573,3557,3546,3550,3570,3548,3552,3556,3565,3551,3553,3567,3560,3559,3554,3558,3547,3555,3566,3563,3564,1407,1887,1415,1895,1404,1884,1408,1888,1406,1886,1410,1890,1414,1894,1409,1889,1411,1891,1418,1898,1417,1897,1412,1892,1416,1896,1405,1885,1413,1893,5348,1902,1910,1899,1903,1901,1905,1909,1904,1906,1913,1912,1907,1911,1900,1908,1400,5364,1917,1925,1914,1918,1916,1920,1924,1919,1921,1928,1927,1922,1926,1915,1923,3600,3591,3603,3590,3597,3578,3598,3601,3602,3586,3575,3579,3599,3577,3581,3585,3594,3580,3582,3596,3589,3588,3583,3587,3576,3584,3595,3592,3593,39957,39969,39954,39962,39970,39963,39982,39961,39956,39955,39958,39974,39985,39981,39960,39959,39983,39979,39975,39966,39980,39978,39964,39971,39972,39977,39973,39984,39976,39965,39968,39967,40053,40065,40050,40058,40066,40059,40078,40057,40052,40051,40054,40070,40081,40077,40056,40055,40079,40075,40071,40062,40076,40074,40060,40067,40068,40073,40069,40080,40072,40061,40064,40063,39737,39725,39740,39732,39724,39731,39822,39733,39738,39739,39736,39720,39825,39821,39734,39735,39823,39819,39719,39728,39820,39716,39730,39723,39722,39717,39721,39824,39718,39729,39726,39727,40213,40225,40210,40218,40226,40219,40238,40217,40212,40211,40214,40230,40241,40237,40216,40215,40239,40235,40231,40222,40236,40234,40220,40227,40228,40233,40229,40240,40232,40221,40224,40223,39829,39841,39826,39834,39842,39835,39854,39833,39828,39827,39830,39846,39857,39853,39832,39831,39855,39851,39847,39838,39852,39850,39836,39843,39844,39849,39845,39856,39848,39837,39840,39839,40117,40129,40114,40122,40130,40123,40142,40121,40116,40115,40118,40134,40145,40141,40120,40119,40143,40139,40135,40126,40140,40138,40124,40131,40132,40137,40133,40144,40136,40125,40128,40127,40085,40097,40082,40090,40098,40091,40110,40089,40084,40083,40086,40102,40113,40109,40088,40087,40111,40107,40103,40094,40108,40106,40092,40099,40100,40105,40101,40112,40104,40093,40096,40095,40277,40289,40274,40282,40290,40283,40302,40281,40276,40275,40278,40294,40305,40301,40280,40279,40303,40299,40295,40286,40300,40298,40284,40291,40292,40297,40293,40304,40296,40285,40288,40287,40149,40161,40146,40154,40162,40155,40174,40153,40148,40147,40150,40166,40177,40173,40152,40151,40175,40171,40167,40158,40172,40170,40156,40163,40164,40169,40165,40176,40168,40157,40160,40159,39861,39873,39858,39866,39874,39867,39886,39865,39860,39859,39862,39878,39889,39885,39864,39863,39887,39883,39879,39870,39884,39882,39868,39875,39876,39881,39877,39888,39880,39869,39872,39871,39769,39757,39772,39764,39756,39763,39744,39765,39770,39771,39768,39752,39741,39745,39766,39767,39743,39747,39751,39760,39746,39748,39762,39755,39754,39749,39753,39742,39750,39761,39758,39759,39808,39816,39805,39809,39807,39811,39815,39810,39812,39818,39813,39817,39806,39814,40181,40193,40178,40186,40194,40187,40206,40185,40180,40179,40182,40198,40209,40205,40184,40183,40207,40203,40199,40190,40204,40202,40188,40195,40196,40201,40197,40208,40200,40189,40192,40191,39801,39789,39804,39796,39788,39795,39776,39797,39802,39803,39800,39784,39773,39777,39798,39799,39775,39779,39783,39792,39778,39780,39794,39787,39786,39781,39785,39774,39782,39793,39790,39791,39989,40001,39986,39994,40002,39995,40014,39993,39988,39987,39990,40006,40017,40013,39992,39991,40015,40011,40007,39998,40012,40010,39996,40003,40004,40009,40005,40016,40008,39997,40000,39999,40309,40321,40306,40314,40322,40315,40334,40313,40308,40307,40310,40326,40336,40333,40312,40311,40335,40331,40327,40318,40332,40330,40316,40323,40324,40329,40325,40337,40328,40317,40320,40319,39925,39937,39922,39930,39938,39931,39950,39929,39924,39923,39926,39942,39953,39949,39928,39927,39951,39947,39943,39934,39948,39946,39932,39939,39940,39945,39941,39952,39944,39933,39936,39935,39893,39905,39890,39898,39906,39899,39918,39897,39892,39891,39894,39910,39921,39917,39896,39895,39919,39915,39911,39902,39916,39914,39900,39907,39908,39913,39909,39920,39912,39901,39904,39903,40021,40033,40018,40026,40034,40027,40046,40025,40020,40019,40022,40038,40049,40045,40024,40023,40047,40043,40039,40030,40044,40042,40028,40035,40036,40041,40037,40048,40040,40029,40032,40031,40245,40257,40242,40250,40258,40251,40270,40249,40244,40243,40246,40262,40273,40269,40248,40247,40271,40267,40263,40254,40268,40266,40252,40259,40260,40265,40261,40272,40264,40253,40256,40255,1389,5375,39714,1393,5371,1932,1940,1929,1933,1931,1935,1939,1934,1936,1943,1942,1937,1941,1930,1938,1947,1955,1944,1948,1946,1950,1954,1949,1951,1958,1957,1952,1956,1945,1953,5350,5349,1391,5373,1977,1985,1974,1978,1976,1980,1984,1962,1970,1959,1963,1961,1965,1969,1964,1966,1973,1972,1967,1971,1960,1968,3629,3620,3632,3619,3626,3607,3627,3630,3631,3615,3604,3608,3628,3606,3610,3614,3623,3609,3611,3625,3618,3617,3612,3616,3605,3613,3624,3621,3622,1979,1981,1988,1987,1982,1986,1975,1983,3658,3649,3661,3648,3655,3636,3656,3659,3660,3644,3633,3637,3657,3635,3639,3643,3652,3638,3640,3654,3647,3646,3641,3645,3634,3642,3653,3650,3651,1395,5369,1399,5365,1992,2000,1989,1993,1991,1995,1999,1994,1996,2003,2002,1997,2001,1990,1998,3687,3678,3690,3677,3684,3665,3685,3688,3689,3673,3662,3666,3686,3664,3668,3672,3681,3667,3669,3683,3676,3675,3670,3674,3663,3671,3682,3679,3680,2007,2015,2004,2008,2006,2010,2014,2009,2011,2018,2017,2012,2016,2005,2013,3716,3707,3719,3706,3713,3694,3714,3717,3718,3702,3691,3695,3715,3693,3697,3701,2022,2030,2019,2023,2021,2025,2029,2024,2026,2033,2032,2027,2031,2020,2028,3710,3696,3698,3712,3705,3704,3699,3703,3692,3700,3711,3708,3709,42081,3745,3736,3748,3735,3742,3723,3743,3746,3747,3731,3720,3724,3744,3722,3726,3730,3739,3725,3727,3741,3734,3733,3728,3732,3721,3729,3740,3737,3738,3774,3765,3777,3764,3771,3752,3772,3775,3776,3760,3749,3753,3773,3751,3755,3759,3768,3754,3756,3770,3763,3762,3757,3761,3750,3758,3769,3766,3767,5315,5327,5312,5320,5328,5321,5340,5319,5314,5313,5316,5332,5342,5339,5318,5317,5341,5337,5333,3803,3794,3806,3793,3800,3781,3801,3804,3805,3789,3778,3782,3802,3780,3784,3788,3797,3783,3785,3799,3792,3791,3786,3790,3779,3787,3798,3795,3796,3832,3823,3835,3822,3829,3810,3830,3833,3834,3818,3807,3811,3831,3809,3813,3817,3826,3812,3814,3828,3821,3820,3815,3819,3808,3816,3827,3824,3825,5324,5338,5336,5322,5329,5330,5335,5331,5343,5334,5323,5326,5325,5356,1394,5370,1396,5368,5354,1403,5360,1402,5362,1397,5367,3861,3852,3864,3851,3858,3839,3859,3862,3863,3847,3836,3840,3860,3838,3842,3846,3855,3841,3843,3857,3850,3849,3844,3848,3837,3845,3856,3853,3854,1401,5363,1390,5374,1398,5366,5355,5358,5357,2037,2045,2034,2038,2036,2040,2044,2039,2041,2048,2047,2042,2046,2035,2043,3890,3881,3893,3880,3887,3868,3888,3891,3892,3876,3865,3869,3889,3867,3871,3875,3884,3870,3872,3886,3879,3878,3873,3877,3866,3874,3885,3882,3883,3919,3910,3922,3909,3916,3897,3917,3920,3921,3905,3894,3898,3918,3896,3900,3904,3913,3899,3901,3915,3908,3907,3902,3906,3895,3903,3914,3911,3912,4347,4359,4344,4350,4360,4354,4372,4351,4348,4346,4352,4364,4375,4371,4349,4373,4369,4365,4317,4327,4315,4321,4329,4322,4342,4318,4316,4314,4320,4334,4345,4339,4319,4341,4338,4333,4325,4340,4336,4323,4331,4330,4337,4332,4343,4335,4324,4328,4326,4287,4297,4282,4290,4298,4291,4310,4288,4284,4285,4289,4301,4313,4309,4286,4311,4307,4303,4294,4308,4306,4292,4299,4300,4305,4302,4312,4304,4293,4296,4295,4356,4370,4368,4353,4361,4362,4367,4363,4374,4366,4355,4358,4357,39713,4255,4265,4253,4257,4267,4261,4278,4258,4254,4252,4259,4271,4283,4279,4256,4280,4275,4273,4263,4277,4276,4260,4268,4270,4274,4269,4281,4272,4262,4266,4264,4224,4235,4221,4227,4236,4229,4248,4226,4223,4222,4228,4240,4251,4246,4225,4249,4244,4241,4232,4247,4245,4230,4237,4238,4243,4239,4250,4242,4231,4234,4233,4193,4204,4190,4196,4206,4198,4217,4195,4192,4191,4197,4210,4220,4216,4194,4218,4213,4209,4202,4215,4214,4199,4205,4207,4211,4208,4219,4212,4200,4203,4201,4163,4173,4159,4165,4174,4167,4187,4164,4161,4160,4166,4178,4189,4185,4162,4186,4183,4179,4131,4142,4128,4134,4143,4136,4155,4133,4130,4129,4135,4147,4158,4154,4132,4156,4152,4148,4139,4153,4151,4137,4144,4145,4150,4146,4157,4149,4138,4141,4140,4170,4184,4182,4168,4175,4176,4181,4177,4188,4180,4169,4172,4171,4100,4111,4097,4103,4112,4105,4124,4102,4099,4098,4104,4116,4126,4123,4101,4125,4121,4117,4108,4122,4120,4106,4113,4114,4119,4115,4127,4118,4107,4110,4109,3948,3939,3951,3938,3945,3926,3946,3949,3950,3934,3923,3927,3947,3925,3929,3933,3942,3928,3930,3944,3937,3936,3931,3935,3924,3932,3943,3940,3941,3977,3968,3980,3967,3974,3955,3975,3978,3979,3963,3952,3956,3976,3954,3958,3962,3971,3957,3959,3973,3966,3965,3960,3964,3953,3961,3972,3969,3970,4006,3997,4009,3996,4003,3984,4004,4007,4008,3992,3981,3985,4005,3983,3987,3991,4000,3986,3988,4002,3995,3994,3989,3993,3982,3990,4001,3998,3999,1242,1250,1239,1243,1241,1245,1249,1244,1246,1253,1252,1247,1251,1240,1248,1257,1265,1254,1258,1256,1260,1264,1259,1261,1268,1267,1262,1266,1255,1263,1272,1280,1269,1273,1271,1275,1279,1274,1276,1283,1282,1277,1281,1270,1278,4035,4026,4038,4025,4032,4013,4033,4036,4037,4021,4010,4014,4034,4012,4016,4020,4029,4015,4017,4031,4024,4023,4018,4022,4011,4019,4030,4027,4028,4064,4055,4067,4054,4061,4042,4062,4065,4066,4050,4039,4043,4063,4041,4045,4049,4058,4044,4046,4060,4053,4052,4047,4051,4040,4048,4059,4056,4057,4093,4084,4096,4083,4090,4071,4091,4094,4095,4079,4068,4072,4092,4070,4074,4078,4087,4073,4075,4089,4082,4081,4076,4080,4069,4077,4088,4085,4086,5283,5295,5280,5288,5296,5289,5308,5287,5282,5281,5284,5300,5311,5306,5286,5285,5309,5305,5301,5292,5307,5304,5290,5297,5298,5303,5299,5310,5302,5291,5294,5293,42019,39712,1189,1181,1132,1133,1192,1188,1190,1186,1182,1187,1185,1178,1179,1184,1180,1191,1183,39710,39711,681,256,18,28)
Function: update_meta_cache()

@wonderboymusic19 months ago

comment:7 @wonderboymusic19 months ago

  • Keywords has-patch added; reporter-feedback dev-feedback removed
  • Milestone changed from Awaiting Review to 3.7

Here's my take on this:

  1. You never want to arbitrarily update the meta cache for a billion posts at once, you only want to cache data as its read or as a page's bundle of posts is loaded. If you are loading 30,000 posts, chances are, you are only loading them once for something like an import. If your traffic is bursty, you are probably only having a subset of posts read at scale
  2. No function that accepts an arbitrary number of $object_ids that are passed to update_meta_cache() ever reads the return value of the function, so there's no downside to it returning false to skip the billion queries and cache adds
  3. When this function is called with any number of $object_ids greater than the filtered value of meta_cache_limit, we should just bail.

My patch provides the meta_cache_limit filter and sets the default to 1000

comment:8 @nacin19 months ago

The limit is not bad but it is tough to get around it when you want to. Either a $force = false parameter, or have update_meta_cache() that wraps a force_update_meta_cache() that ignores any limit.

You have to think that if someone is doing an unbounded query on massive amounts of data (deliberately), they should know to turn off the cache priming.

Or: maybe we should chunk this. If we get greater than 200 IDs, then break them into groups of 100 and do individual queries. That's better than timing out, but at the same time, maybe follows the spirit of cache-priming a bit better than not priming the cache at all.

comment:9 @nacin17 months ago

  • Keywords 2nd-opinion added

Not sure what our direction should be here. Any opinions?

comment:10 @nacin17 months ago

  • Milestone changed from 3.7 to 3.8

comment:11 @wonderboymusic15 months ago

  • Milestone changed from 3.8 to Future Release

comment:12 @GaryJ7 months ago

I've got a meal plan CPT, where each meal is linked to multiple recipe CPTs (and that in turn, to multiple ingredient CPTs) with Posts 2 Posts plugin. When trying to show the list of meal plans (edit.php?post_type=mp_mealplan), I'm getting a fatal error due to memory allocation. It points to https://core.trac.wordpress.org/browser/tags/3.9.1/src/wp-includes/meta.php#L747 as being the initial issue (array mapping of intval over all object IDs). Hacking WP to echo a count of $object_ids gives a value of 13847 objects for the instance where the fatal error happens.

There are 2600+ meal plans, each with a guestimate average of 6-10 recipes. Each recipe has 6-10 ingredients on average.

It's on Synthesis hosting, with W3TC (the wp-content/advanced-cache.php references W3TC too).

I can temporarily get around it by bumping the WP memory constants to 512M, but if there was a chunking option, or a way to smooth out or avoid this caching, then it would definitely help the extreme cases.

comment:13 @jipmoors5 weeks ago

It should be made clear (on the codex) that this functionality is not intended to cram up everything you have because of the unloading of cache when the memory is full.

Though I think that batching up makes a lot more sense compared to limiting the number of cache items the system allows. I don't know what kind of system this will run on or how huge the site has become.

Discouraging would be the main goal, but we don't want to break sites or scare people away, I guess.

@jipmoors5 weeks ago

Batching meta ids when updating cache

comment:14 @jipmoors5 weeks ago

  • Keywords needs-codex added

comment:15 @jipmoors5 weeks ago

As for the batch size I considered a couple of things; since meta_key is 255 varchar this is a size that we can calculate with, meta_value can be 4GB, so calculating is fairly useless.

Perhaps the memory_limit could be used to set a more dynamic baseline, but by adding the filter option I thought it best to leave it at a relative low value and let caching plugins or users decide for themselves if they want to modify this to minimise the multidimensional overhead added.

Thoughts are welcome of course.

Note: See TracTickets for help on using tickets.