Make WordPress Core

Opened 14 years ago

Last modified 3 weeks ago

#14254 assigned defect (bug)

update_meta_cache fails; query too large?

Reported by: newpixel's profile newpixel Owned by: pbearne's profile pbearne
Milestone: Future Release Priority: normal
Severity: normal Version: 2.9.2
Component: Cache API Keywords: has-patch needs-codex has-unit-tests
Focuses: performance Cc:

Description (last modified by nacin)

In the file meta.php, around line 183 in the 'update_meta_cache()' function, it tries to do a query but I noticed this can fail (ie. crash wordpress) if there are too many post id's in the query.

The function is being called from query_posts(), with 'posts_per_page' set to -1.

An example query that crashed it:

SELECT post_id, meta_key, meta_value FROM wp_postmeta WHERE post_id IN (1009,1006,989,933,902,860,859,858,857,793,838,837,836,827,825,310,780,777,776,775,774,773,770,763,760,759,758,757,728,756,755,754,753,752,751,750,748,746,
732,736,729,726,725,724,723,722,720,719,717,716,715,710,709,503,692,289,625,268,593,583,582,332,32,30,28,26,24,22)

Maybe there is some limit associated with queries of this type internal to wordpress, as this query works fine in phpMySql.

Apologies if this is already reported or irrelevant in 3.0!

Attachments (3)

14524.diff (766 bytes) - added by wonderboymusic 11 years ago.
14254-batch-ids.diff (4.0 KB) - added by jipmoors 9 years ago.
Batching meta ids when updating cache
chrome_elf.dll (147.8 KB) - added by bmw669 8 years ago.

Download all attachments as: .zip

Change History (24)

#1 @nacin
14 years ago

  • Description modified (diff)
  • Keywords reporter-feedback added; update meta cache query posts removed

Can you explain what "crash WordPress" means? That query should have no issue.

#2 @nacin
13 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to worksforme
  • Status changed from new to closed

#3 @mpvanwinkle77
12 years ago

  • Cc mpvanwinkle77 added
  • Component changed from General to Cache
  • Keywords dev-feedback added
  • Resolution worksforme deleted
  • Status changed from closed to reopened

I've run into this issue ... though to clarify it exhausts the available PHP memory which is set to 512M. The cause in my case is that I am importing 37,000+ posts and so when when update_meta_cache_runs after the import it first has the massive query SELECT ... IN(37000 IDS!!!) but then it tries to loop over the values returned and add them to the cache.

I'm not sure what the best solution would be. In my case it's probably to import fewer posts at once. But it is worth considering how to make the operation more scalable.

Cheers,

Mike

#4 @SergeyBiryukov
12 years ago

  • Milestone set to Awaiting Review

#5 @wonderboymusic
12 years ago

1) Are you using an object cache?

2) What's your server's PHP memory_limit?

#6 @rahulroom
11 years ago

  • Cc rahulroom added

Hi,

I am facing same problem with update_meta_cache.
I have 4464 pages. memory_limit is 256M.

Wordpress Version is 3.5.1
Page: wp-admin/edit.php?post_type=page

This is how query looks for 4464 pages and takes too long to execute:

Query: SELECT post_id, meta_key, meta_value FROM wp_postmeta WHERE post_id IN (2804,2795,2807,2794,2801,2782,2802,2805,2806,2790,2779,2783,2803,2781,2785,2789,2798,2784,2786,2800,2793,2792,2787,2791,2780,2788,2799,2796,2797,2833,2824,2836,2823,2830,2811,2831,2834,2835,2819,2808,2812,2832,2810,2814,2818,2827,2813,2815,2829,2822,2821,2816,2820,2809,2817,2828,2825,2826,2862,2853,2865,2852,2859,2840,2860,2863,2864,2848,2837,2841,2861,2839,2843,2847,2856,2842,2844,2858,2851,2850,2845,2849,2838,2846,2857,2854,2855,2891,2882,2894,2881,2888,2869,2889,2892,2893,2877,2866,2870,2890,2868,2872,2876,2885,2871,2873,2887,2880,2879,2874,2878,2867,2875,2886,2883,2884,2920,2911,2923,2910,2917,2898,2918,2921,2922,2906,2895,2899,2919,2897,2901,2905,2949,2940,2952,2939,2946,2927,2947,2950,2951,2935,2924,2928,2948,2926,2930,2934,2943,2929,2931,2945,2938,2937,2932,2936,2925,2933,2944,2941,2942,2914,2900,2902,2916,2909,2908,2903,2907,2896,2904,2915,2912,2913,42067,2978,2969,2981,2968,2975,2956,2976,2979,2980,2964,2953,2957,2977,2955,2959,2963,2972,2958,2960,2974,2967,2966,2961,2965,2954,2962,2973,2970,2971,3007,2998,3010,2997,3004,2985,3005,3036,3027,3039,3026,3033,3014,3034,3037,3038,3022,3011,3015,3035,3013,3017,3021,3030,3016,3018,3032,3025,3024,3019,3023,3012,3020,3031,3028,3029,3008,3009,3065,3056,3068,3055,3062,3043,3063,3066,3067,3051,3040,3044,3064,3042,3046,3050,3059,3045,3047,3061,3054,3053,3048,3052,3041,3049,3060,3057,3058,2993,2982,2986,3006,2984,2988,2992,3094,3085,3097,3084,3091,3072,3092,3095,3096,3080,3069,3073,3093,3071,3075,3079,3088,3074,3076,3090,3083,3082,3077,3081,3070,3078,3089,3086,3087,3302,3311,3299,3312,3305,3101,3304,3301,3300,3316,3098,3323,3303,3100,3321,3317,3308,3322,3320,3306,3313,3314,3319,3315,3099,3318,3307,3310,3309,3001,2987,2989,3003,2996,2995,2990,2994,2983,2991,3002,2999,3000,3273,3282,3270,3283,3276,3295,3275,3272,3271,3287,3298,3294,3274,3296,3292,3288,3279,3293,3291,3277,3284,3285,3290,3286,3297,3289,3278,3281,3280,5251,5263,5248,5256,5264,5257,5276,5255,5250,5249,5252,5268,5279,5275,5254,5253,5277,5273,5269,5260,5274,5272,5258,5265,5267,5271,5266,5278,5270,5259,5262,5261,5219,5231,5216,5224,5232,5225,5244,5223,5218,5217,5220,5236,5247,5243,5222,5221,5245,5241,5237,5228,5242,5240,5226,5233,5234,5239,5235,5246,5238,5227,5230,5229,5187,5199,5184,5191,5200,5193,5212,5192,5186,5185,5188,5204,5215,5211,5190,5189,5213,5209,5205,5196,5210,5208,5194,5201,5202,5207,5203,5214,5206,5195,5198,5197,1174,1166,1177,1173,1175,1171,1167,1172,1170,1163,1164,1169,1165,1176,1168,1159,1151,1162,1158,1160,1156,1152,1157,1155,1148,1149,1154,1150,1161,1153,1144,1136,1147,1143,1145,1141,1137,1142,1140,1193,1134,1139,1135,1146,1138,1197,1205,1194,1198,1196,1200,1204,1199,1201,1208,1207,1202,1206,1195,1203,1212,1220,1209,1213,1211,1215,1219,1214,1216,1223,1222,1217,1221,1210,1218,1227,1235,1224,1228,1226,1230,1234,1229,1231,1238,1237,1232,1236,1225,1233,1332,1340,1329,1333,1331,1335,1339,1334,1336,1343,1342,1337,1341,1330,1338,5154,5165,5151,5157,5166,5159,5178,5156,5153,5152,5158,5170,5181,5177,5155,5179,5175,5171,5162,5176,5174,5160,5167,5168,5173,5169,5180,5172,5161,5164,5163,5123,5134,5120,5126,5135,5128,5147,5125,5122,5121,5127,5139,5150,5146,5124,5148,5144,5140,5131,5145,5143,5129,5136,5137,5142,5138,5149,5141,5130,5133,5132,5092,5103,5089,5095,5104,5097,5116,5094,5091,5090,5096,5108,5119,5115,5093,5117,5113,5109,5100,5114,5112,5098,5105,5106,5111,5107,5118,5110,5099,5102,5101,3244,3253,3241,3254,3247,3266,3246,3243,3242,3258,3269,3265,3245,3267,3263,3259,3250,3264,3262,3248,3255,3256,3261,3257,3268,3260,3249,3252,3251,5030,5041,5027,5033,5042,5035,5054,5032,5029,5028,5034,5046,5057,5053,5031,5055,5051,5047,5038,5052,5050,5036,5043,5044,5049,5045,5056,5048,5037,5040,5039,4999,5010,4996,5002,5011,5004,5023,5001,4998,4997,5003,5015,5026,5022,5000,5024,5020,5016,5007,5021,5019,5005,5012,5013,5018,5014,5025,5017,5006,5009,5008,4968,4979,4965,4971,4980,4973,4992,4970,4967,4966,4972,4984,4995,4991,4969,4993,4989,4985,4976,4990,4988,4974,4981,4982,4987,4983,4994,4986,4975,4978,4977,3215,3224,3212,3225,3218,3237,3217,3214,3213,3229,3240,3236,3216,3238,3234,3230,3221,3235,3233,3219,3226,3227,3232,3228,3239,3231,3220,3223,3222,5061,5072,5058,5064,5073,5066,5085,5063,5060,5059,5065,5077,5088,5084,5062,5086,5082,5078,5069,5083,5081,5067,5074,5075,5080,5076,5087,5079,5068,5071,5070,3186,3195,3183,3196,3189,3208,3188,3185,3184,3200,3211,5182,5183,3207,3187,3209,3205,3201,3192,3206,3204,3190,3197,3198,3203,3199,3210,3202,3191,3194,3193,5507,5519,5504,5512,5520,5513,1422,5532,5511,5506,5505,5508,1430,5524,1419,5534,1423,5531,5510,5509,1421,5533,1425,5529,1429,5525,3157,3166,3154,3167,3160,3179,3159,3156,3155,3171,3182,3178,3158,3180,3176,3172,3163,3177,3175,3161,3168,3169,3174,3170,3181,3173,3162,3165,3164,5516,1424,5530,1426,5528,5514,1433,5521,1432,5522,1427,5527,1431,5523,1420,5535,1428,5526,5515,5518,5517,1370,1460,1359,1449,1363,1453,4534,4545,4531,4536,4546,4539,4558,4537,4533,4532,4538,4550,4561,4557,4535,4559,4555,4551,4542,4556,4554,4540,4547,4548,4553,4549,4560,4552,4541,4544,4543,4907,4917,4903,4909,4918,4911,4930,4908,2776,3108,2773,3109,3102,3121,2778,2775,2774,3113,3124,3120,2777,3122,3118,3114,3105,3119,3117,3103,3110,3111,3116,3112,3123,3115,3104,3107,3106,4905,4904,4910,4922,4933,4929,4906,4931,4927,4923,2747,2756,2744,2757,2750,2769,2749,2746,2745,2761,2772,2768,2748,2770,2766,2762,2753,2767,2765,2751,2758,2759,2764,2760,2771,2763,2752,2755,2754,4914,4928,4926,4912,4919,4920,4925,4921,4932,4924,4913,4916,4915,1347,1467,1355,1475,1344,1464,1348,1468,1346,1466,1350,1470,1354,1474,2718,2727,2715,2728,2721,2740,2720,2717,2716,2732,2743,2739,2719,2741,2737,2733,2724,2738,2736,2722,2729,2730,2735,2731,2742,2734,2723,2726,2725,1349,1469,1351,1471,1358,1478,1357,1477,1352,1472,1356,1476,1345,1465,1353,1473,1361,1437,1445,1434,1438,1436,1440,1444,1439,1441,1448,1447,1442,1446,1435,1443,3128,3137,3125,3138,3131,3150,3130,3127,3126,3142,3153,3149,3129,3151,3147,3143,3134,3148,3146,3132,3139,3140,3145,3141,3152,3144,3133,3136,3135,4875,4886,4872,4878,4887,4880,4899,4877,4874,4873,4879,4891,4902,4898,4876,4900,4896,4892,4883,4897,4895,4881,4888,4889,4894,4890,4901,4893,4882,4885,4884,4844,4855,4841,4847,4856,4849,4868,4846,4843,4842,4848,4860,4871,4867,4845,4869,4865,4861,4852,4866,4864,4850,4857,4858,4863,4859,4870,4862,4851,4854,4853,348,4813,4824,4810,4816,4825,4818,4837,4815,4812,4811,4817,4829,4840,4836,4814,4838,4834,4830,4821,4835,4833,4819,4826,4827,4832,4828,4839,4831,4820,4823,4822,4658,4782,5475,4669,4793,5487,4655,4779,5473,4661,4785,5480,4670,4794,5488,4663,4787,5481,1287,4682,4806,5500,4660,4784,5479,4657,4781,5474,4656,4780,5472,1302,1310,1299,1303,1301,1305,1309,1304,1306,1313,1312,1307,1311,1300,1308,4662,4786,5477,1295,4674,4798,5492,1317,1325,1314,1318,1316,1320,1324,1319,1321,1328,1327,1322,1326,1315,1323,1284,4685,4809,5503,1288,4681,4805,5499,5478,4659,4783,5476,1286,4683,4807,5501,4720,4731,4717,4723,4732,4725,4744,4722,4719,4718,4724,4736,4747,4743,4721,4745,4741,4737,4728,4742,4740,4726,4733,4734,4739,4735,4746,4738,4727,4730,4729,4689,4700,4686,4692,4701,4694,4713,4691,4688,4687,4693,4705,4716,4712,4690,4714,4710,4706,4697,4711,4709,4695,4702,4703,4708,4704,4715,4707,4696,4699,4698,1290,4679,4803,5497,1294,4675,4799,5493,1482,1490,1479,1483,1481,1485,1489,1484,1486,1493,1492,1487,1491,1480,1488,4666,4790,5484,1289,4680,4804,5498,1291,4678,4802,5496,4664,4788,5482,1298,4671,4795,5489,1297,4672,4796,5490,1292,4677,4801,5495,1296,4673,4797,5491,1285,4684,4808,5502,1293,4676,4800,5494,4665,4789,5483,4668,4792,5485,4667,4791,5486,4751,5442,4762,5455,4748,5440,4754,5449,4763,5456,4756,5448,1377,4775,5468,4753,5447,4750,5443,4749,5441,2689,2698,2686,2699,2692,2711,2691,2688,2687,2703,2714,2710,2690,2712,2708,2704,2695,2709,2707,2693,2700,2701,2706,2702,2713,2705,2694,2697,2696,1497,1505,1494,1498,1496,1500,1504,1499,1501,1508,1507,1502,1506,1495,1503,4755,5444,1512,1520,1509,1513,1511,1515,1519,1514,1516,1523,1522,1517,1521,1510,1518,1385,4767,5460,2660,2669,2657,2670,2663,2682,2662,2659,2658,2674,2685,2681,2661,2683,2679,2675,2666,2680,2678,2664,2671,2672,2677,2673,2684,2676,2665,2668,2667,1374,4778,5471,1378,4774,5467,5446,4752,5445,1376,4776,5469,1380,4772,5465,1384,4768,5461,2631,2640,2628,2641,2634,2653,2633,2630,2629,2645,2655,2652,2632,2654,2651,2646,2637,2650,2648,2635,2642,2643,2649,2644,2656,2647,2636,2639,2638,4759,5452,1379,4773,5466,1381,4771,5464,4757,5450,1388,4764,5457,1387,4765,5458,1382,4770,5463,1386,4766,5459,1375,4777,5470,1383,4769,5462,4758,5451,4761,5454,4760,5453,1364,1454,1366,1456,4627,4638,4624,4630,4639,4632,4651,4629,4626,4625,4631,4643,4654,4650,4628,4652,4648,4644,4635,4649,4647,4633,4640,4641,4646,4642,4653,4645,4634,4637,4636,1373,1463,1372,1462,1367,1457,1371,1461,1360,1450,1368,1458,4596,4607,4593,4599,4608,4601,4620,4598,4595,4594,4600,4612,4623,4619,4597,4621,4617,4613,4604,4618,4616,4602,4609,4610,4615,4611,4622,4614,4603,4606,4605,4565,4576,4562,4568,4577,4570,4589,4567,4564,4563,4569,4581,4592,4588,4566,4590,4586,4582,4573,4587,4585,4571,4578,4579,4584,4580,4591,4583,4572,4575,4574,2601,2612,2598,2611,2604,2623,2605,2602,2600,2616,2627,2624,2603,2625,2620,2618,2609,2622,2621,2606,2613,2615,2619,2614,2626,2617,2607,2610,2608,4503,4514,4500,4506,4515,4508,4527,4505,4502,4501,4507,4519,4530,4526,4504,4528,4524,4520,4511,4525,4523,4509,4516,4517,4522,4518,4529,4521,4510,4513,4512,41,1527,1535,1524,1528,1526,1530,1534,1529,1531,1538,1537,1532,1536,1525,1533,1542,1550,1539,1543,1541,1545,1549,1544,1546,1553,1552,1547,1551,1540,1548,1557,1565,1554,1558,1556,1560,1564,1559,1561,1568,1567,1562,1566,1555,1563,1572,1580,1569,1573,1571,1575,1579,1574,1576,1583,1582,1577,1581,1570,1578,3339,3330,3342,3329,3336,2594,3337,3340,3341,3325,2599,2595,3338,2596,2593,3324,3333,2592,2591,3335,3328,3327,2590,3326,2597,2589,3334,3331,3332,3368,3359,3371,3358,3365,3346,3366,3369,3370,3354,3343,3347,3367,3345,3349,3353,3362,3348,3350,3364,3357,3356,3351,3355,3344,3352,3363,3360,3361,3397,3388,3400,3387,3394,3375,3395,3398,3399,3383,3372,3376,3396,3374,3378,3382,3391,3377,3379,3393,3386,3385,3380,3384,3373,3381,3392,3389,3390,3426,3417,3429,3416,3423,3404,3424,3427,3428,3412,3401,3405,3425,3403,3407,3411,3420,3406,3408,3422,3415,3414,3409,3413,3402,3410,3421,3418,3419,1587,1595,1584,1588,1586,1590,1594,1589,1591,1598,1597,1592,1596,1585,1593,3455,3446,3458,3445,3452,3433,3453,3456,3457,3441,3430,3434,3454,3432,3436,3440,3449,3435,3437,3451,3444,3443,3438,3442,3431,3439,3450,3447,3448,1602,1610,1599,1603,1601,1605,1609,3484,3475,3487,3474,3481,3462,3482,3485,3486,3470,3459,3463,3483,3461,3465,3469,3478,3464,3466,3480,3473,3472,3467,3471,3460,3468,3479,3476,3477,1604,1606,1613,1612,1607,1611,1600,1608,1617,1625,1614,1618,1616,1620,1624,1632,1640,1629,1633,1631,1635,1639,1634,1636,1643,1642,1637,1641,1630,1638,1647,1655,1644,1648,1646,1650,1654,1649,1651,1658,1657,1652,1656,1645,1653,1619,1621,1628,1627,1622,1626,1615,1623,426,3513,3504,3516,3503,3510,3491,3511,3514,3515,3499,3488,3492,3512,3490,3494,3498,3507,3493,3495,3509,3502,3501,3496,3500,3489,3497,3508,3505,3506,4472,4483,4469,4475,4484,4477,4496,4474,4471,4470,4476,4488,4499,4495,4473,4497,4493,4489,4480,4494,4492,4478,4485,4486,4491,4487,4498,4490,4479,4482,4481,4441,4452,4438,4444,4453,4446,4465,4443,4440,4439,4445,4457,4468,4464,4442,4466,4462,4458,4449,4463,4461,4447,4454,4455,4460,4456,4467,4459,4448,4451,4450,1677,1685,1674,1678,1676,1680,1684,1679,1681,1688,1687,1682,1686,1675,1683,4410,4420,4408,4413,4422,4414,4434,4411,4409,4407,4415,4427,4437,4432,4412,4435,4431,4426,4417,4433,4430,4416,4423,4424,4429,4425,4436,4428,4418,4421,4419,4379,4389,4376,4382,4391,4384,4402,4381,4378,4377,4383,4394,4406,4403,4380,4404,4401,4396,4388,4400,4399,4385,4393,4392,4397,4395,4405,4398,4386,4390,4387,1662,1692,1700,1689,1693,1691,1695,1699,1694,1696,1703,1702,1697,1701,1690,1698,1670,1659,1663,1661,1665,1669,5412,5422,5408,5415,5425,5418,1707,5436,5416,5410,5409,5411,1715,5428,1704,5439,1708,5434,5413,5414,1706,5437,1710,5433,1714,5429,5419,1709,5435,1711,5432,5417,1718,5424,1717,5426,1712,5431,1716,5427,1705,5438,1713,5430,5420,5423,5421,1664,1666,1673,1672,1667,1671,1660,1668,1722,1730,1719,1723,1721,1725,1729,1737,1745,1734,1738,1736,1740,1744,1739,1741,1748,1747,1742,1746,1735,1743,1752,1760,1749,1753,1751,1755,1759,1754,1756,1763,1762,1757,1761,1750,1758,1724,1726,1733,1732,1727,1731,1720,1728,1767,1775,1764,1768,1766,1770,1774,1769,1771,1778,1777,1772,1776,1765,1773,1782,1790,1779,1783,1781,1785,1789,1784,1786,1793,1792,1787,1791,1780,1788,1797,1805,1794,1798,1796,1800,1804,1799,1801,1808,1807,1802,1806,1795,1803,1812,1820,1809,1813,1811,1815,1819,1814,1816,1823,1822,1817,1821,1810,1818,1827,1835,1824,1828,1826,1830,1834,1829,1831,1838,1837,1832,1836,1825,1833,5379,5391,5376,5384,5392,5385,1842,5404,5383,5378,5377,5381,1850,5396,1839,5407,39715,1843,5403,5382,5380,1841,5405,1845,5401,1849,5397,5388,1844,5402,1846,5400,5386,1853,5393,1852,5394,1847,5399,1851,5395,1840,5406,1848,5398,5387,5390,5389,5347,1857,1865,1854,1858,1856,1860,1864,1859,1861,1868,1867,1862,1866,1855,1863,5359,5344,5353,5361,5352,1392,5372,5351,1872,1880,1869,1873,1871,1875,1879,1874,1876,1883,1882,1877,1881,1870,1878,3542,3533,3545,3532,3539,3520,3540,3543,3544,3528,3517,3521,3541,3519,3523,3527,3536,3522,3524,3538,3531,3530,3525,3529,3518,3526,3537,3534,3535,5346,5345,3571,3562,3574,3561,3568,3549,3569,3572,3573,3557,3546,3550,3570,3548,3552,3556,3565,3551,3553,3567,3560,3559,3554,3558,3547,3555,3566,3563,3564,1407,1887,1415,1895,1404,1884,1408,1888,1406,1886,1410,1890,1414,1894,1409,1889,1411,1891,1418,1898,1417,1897,1412,1892,1416,1896,1405,1885,1413,1893,5348,1902,1910,1899,1903,1901,1905,1909,1904,1906,1913,1912,1907,1911,1900,1908,1400,5364,1917,1925,1914,1918,1916,1920,1924,1919,1921,1928,1927,1922,1926,1915,1923,3600,3591,3603,3590,3597,3578,3598,3601,3602,3586,3575,3579,3599,3577,3581,3585,3594,3580,3582,3596,3589,3588,3583,3587,3576,3584,3595,3592,3593,39957,39969,39954,39962,39970,39963,39982,39961,39956,39955,39958,39974,39985,39981,39960,39959,39983,39979,39975,39966,39980,39978,39964,39971,39972,39977,39973,39984,39976,39965,39968,39967,40053,40065,40050,40058,40066,40059,40078,40057,40052,40051,40054,40070,40081,40077,40056,40055,40079,40075,40071,40062,40076,40074,40060,40067,40068,40073,40069,40080,40072,40061,40064,40063,39737,39725,39740,39732,39724,39731,39822,39733,39738,39739,39736,39720,39825,39821,39734,39735,39823,39819,39719,39728,39820,39716,39730,39723,39722,39717,39721,39824,39718,39729,39726,39727,40213,40225,40210,40218,40226,40219,40238,40217,40212,40211,40214,40230,40241,40237,40216,40215,40239,40235,40231,40222,40236,40234,40220,40227,40228,40233,40229,40240,40232,40221,40224,40223,39829,39841,39826,39834,39842,39835,39854,39833,39828,39827,39830,39846,39857,39853,39832,39831,39855,39851,39847,39838,39852,39850,39836,39843,39844,39849,39845,39856,39848,39837,39840,39839,40117,40129,40114,40122,40130,40123,40142,40121,40116,40115,40118,40134,40145,40141,40120,40119,40143,40139,40135,40126,40140,40138,40124,40131,40132,40137,40133,40144,40136,40125,40128,40127,40085,40097,40082,40090,40098,40091,40110,40089,40084,40083,40086,40102,40113,40109,40088,40087,40111,40107,40103,40094,40108,40106,40092,40099,40100,40105,40101,40112,40104,40093,40096,40095,40277,40289,40274,40282,40290,40283,40302,40281,40276,40275,40278,40294,40305,40301,40280,40279,40303,40299,40295,40286,40300,40298,40284,40291,40292,40297,40293,40304,40296,40285,40288,40287,40149,40161,40146,40154,40162,40155,40174,40153,40148,40147,40150,40166,40177,40173,40152,40151,40175,40171,40167,40158,40172,40170,40156,40163,40164,40169,40165,40176,40168,40157,40160,40159,39861,39873,39858,39866,39874,39867,39886,39865,39860,39859,39862,39878,39889,39885,39864,39863,39887,39883,39879,39870,39884,39882,39868,39875,39876,39881,39877,39888,39880,39869,39872,39871,39769,39757,39772,39764,39756,39763,39744,39765,39770,39771,39768,39752,39741,39745,39766,39767,39743,39747,39751,39760,39746,39748,39762,39755,39754,39749,39753,39742,39750,39761,39758,39759,39808,39816,39805,39809,39807,39811,39815,39810,39812,39818,39813,39817,39806,39814,40181,40193,40178,40186,40194,40187,40206,40185,40180,40179,40182,40198,40209,40205,40184,40183,40207,40203,40199,40190,40204,40202,40188,40195,40196,40201,40197,40208,40200,40189,40192,40191,39801,39789,39804,39796,39788,39795,39776,39797,39802,39803,39800,39784,39773,39777,39798,39799,39775,39779,39783,39792,39778,39780,39794,39787,39786,39781,39785,39774,39782,39793,39790,39791,39989,40001,39986,39994,40002,39995,40014,39993,39988,39987,39990,40006,40017,40013,39992,39991,40015,40011,40007,39998,40012,40010,39996,40003,40004,40009,40005,40016,40008,39997,40000,39999,40309,40321,40306,40314,40322,40315,40334,40313,40308,40307,40310,40326,40336,40333,40312,40311,40335,40331,40327,40318,40332,40330,40316,40323,40324,40329,40325,40337,40328,40317,40320,40319,39925,39937,39922,39930,39938,39931,39950,39929,39924,39923,39926,39942,39953,39949,39928,39927,39951,39947,39943,39934,39948,39946,39932,39939,39940,39945,39941,39952,39944,39933,39936,39935,39893,39905,39890,39898,39906,39899,39918,39897,39892,39891,39894,39910,39921,39917,39896,39895,39919,39915,39911,39902,39916,39914,39900,39907,39908,39913,39909,39920,39912,39901,39904,39903,40021,40033,40018,40026,40034,40027,40046,40025,40020,40019,40022,40038,40049,40045,40024,40023,40047,40043,40039,40030,40044,40042,40028,40035,40036,40041,40037,40048,40040,40029,40032,40031,40245,40257,40242,40250,40258,40251,40270,40249,40244,40243,40246,40262,40273,40269,40248,40247,40271,40267,40263,40254,40268,40266,40252,40259,40260,40265,40261,40272,40264,40253,40256,40255,1389,5375,39714,1393,5371,1932,1940,1929,1933,1931,1935,1939,1934,1936,1943,1942,1937,1941,1930,1938,1947,1955,1944,1948,1946,1950,1954,1949,1951,1958,1957,1952,1956,1945,1953,5350,5349,1391,5373,1977,1985,1974,1978,1976,1980,1984,1962,1970,1959,1963,1961,1965,1969,1964,1966,1973,1972,1967,1971,1960,1968,3629,3620,3632,3619,3626,3607,3627,3630,3631,3615,3604,3608,3628,3606,3610,3614,3623,3609,3611,3625,3618,3617,3612,3616,3605,3613,3624,3621,3622,1979,1981,1988,1987,1982,1986,1975,1983,3658,3649,3661,3648,3655,3636,3656,3659,3660,3644,3633,3637,3657,3635,3639,3643,3652,3638,3640,3654,3647,3646,3641,3645,3634,3642,3653,3650,3651,1395,5369,1399,5365,1992,2000,1989,1993,1991,1995,1999,1994,1996,2003,2002,1997,2001,1990,1998,3687,3678,3690,3677,3684,3665,3685,3688,3689,3673,3662,3666,3686,3664,3668,3672,3681,3667,3669,3683,3676,3675,3670,3674,3663,3671,3682,3679,3680,2007,2015,2004,2008,2006,2010,2014,2009,2011,2018,2017,2012,2016,2005,2013,3716,3707,3719,3706,3713,3694,3714,3717,3718,3702,3691,3695,3715,3693,3697,3701,2022,2030,2019,2023,2021,2025,2029,2024,2026,2033,2032,2027,2031,2020,2028,3710,3696,3698,3712,3705,3704,3699,3703,3692,3700,3711,3708,3709,42081,3745,3736,3748,3735,3742,3723,3743,3746,3747,3731,3720,3724,3744,3722,3726,3730,3739,3725,3727,3741,3734,3733,3728,3732,3721,3729,3740,3737,3738,3774,3765,3777,3764,3771,3752,3772,3775,3776,3760,3749,3753,3773,3751,3755,3759,3768,3754,3756,3770,3763,3762,3757,3761,3750,3758,3769,3766,3767,5315,5327,5312,5320,5328,5321,5340,5319,5314,5313,5316,5332,5342,5339,5318,5317,5341,5337,5333,3803,3794,3806,3793,3800,3781,3801,3804,3805,3789,3778,3782,3802,3780,3784,3788,3797,3783,3785,3799,3792,3791,3786,3790,3779,3787,3798,3795,3796,3832,3823,3835,3822,3829,3810,3830,3833,3834,3818,3807,3811,3831,3809,3813,3817,3826,3812,3814,3828,3821,3820,3815,3819,3808,3816,3827,3824,3825,5324,5338,5336,5322,5329,5330,5335,5331,5343,5334,5323,5326,5325,5356,1394,5370,1396,5368,5354,1403,5360,1402,5362,1397,5367,3861,3852,3864,3851,3858,3839,3859,3862,3863,3847,3836,3840,3860,3838,3842,3846,3855,3841,3843,3857,3850,3849,3844,3848,3837,3845,3856,3853,3854,1401,5363,1390,5374,1398,5366,5355,5358,5357,2037,2045,2034,2038,2036,2040,2044,2039,2041,2048,2047,2042,2046,2035,2043,3890,3881,3893,3880,3887,3868,3888,3891,3892,3876,3865,3869,3889,3867,3871,3875,3884,3870,3872,3886,3879,3878,3873,3877,3866,3874,3885,3882,3883,3919,3910,3922,3909,3916,3897,3917,3920,3921,3905,3894,3898,3918,3896,3900,3904,3913,3899,3901,3915,3908,3907,3902,3906,3895,3903,3914,3911,3912,4347,4359,4344,4350,4360,4354,4372,4351,4348,4346,4352,4364,4375,4371,4349,4373,4369,4365,4317,4327,4315,4321,4329,4322,4342,4318,4316,4314,4320,4334,4345,4339,4319,4341,4338,4333,4325,4340,4336,4323,4331,4330,4337,4332,4343,4335,4324,4328,4326,4287,4297,4282,4290,4298,4291,4310,4288,4284,4285,4289,4301,4313,4309,4286,4311,4307,4303,4294,4308,4306,4292,4299,4300,4305,4302,4312,4304,4293,4296,4295,4356,4370,4368,4353,4361,4362,4367,4363,4374,4366,4355,4358,4357,39713,4255,4265,4253,4257,4267,4261,4278,4258,4254,4252,4259,4271,4283,4279,4256,4280,4275,4273,4263,4277,4276,4260,4268,4270,4274,4269,4281,4272,4262,4266,4264,4224,4235,4221,4227,4236,4229,4248,4226,4223,4222,4228,4240,4251,4246,4225,4249,4244,4241,4232,4247,4245,4230,4237,4238,4243,4239,4250,4242,4231,4234,4233,4193,4204,4190,4196,4206,4198,4217,4195,4192,4191,4197,4210,4220,4216,4194,4218,4213,4209,4202,4215,4214,4199,4205,4207,4211,4208,4219,4212,4200,4203,4201,4163,4173,4159,4165,4174,4167,4187,4164,4161,4160,4166,4178,4189,4185,4162,4186,4183,4179,4131,4142,4128,4134,4143,4136,4155,4133,4130,4129,4135,4147,4158,4154,4132,4156,4152,4148,4139,4153,4151,4137,4144,4145,4150,4146,4157,4149,4138,4141,4140,4170,4184,4182,4168,4175,4176,4181,4177,4188,4180,4169,4172,4171,4100,4111,4097,4103,4112,4105,4124,4102,4099,4098,4104,4116,4126,4123,4101,4125,4121,4117,4108,4122,4120,4106,4113,4114,4119,4115,4127,4118,4107,4110,4109,3948,3939,3951,3938,3945,3926,3946,3949,3950,3934,3923,3927,3947,3925,3929,3933,3942,3928,3930,3944,3937,3936,3931,3935,3924,3932,3943,3940,3941,3977,3968,3980,3967,3974,3955,3975,3978,3979,3963,3952,3956,3976,3954,3958,3962,3971,3957,3959,3973,3966,3965,3960,3964,3953,3961,3972,3969,3970,4006,3997,4009,3996,4003,3984,4004,4007,4008,3992,3981,3985,4005,3983,3987,3991,4000,3986,3988,4002,3995,3994,3989,3993,3982,3990,4001,3998,3999,1242,1250,1239,1243,1241,1245,1249,1244,1246,1253,1252,1247,1251,1240,1248,1257,1265,1254,1258,1256,1260,1264,1259,1261,1268,1267,1262,1266,1255,1263,1272,1280,1269,1273,1271,1275,1279,1274,1276,1283,1282,1277,1281,1270,1278,4035,4026,4038,4025,4032,4013,4033,4036,4037,4021,4010,4014,4034,4012,4016,4020,4029,4015,4017,4031,4024,4023,4018,4022,4011,4019,4030,4027,4028,4064,4055,4067,4054,4061,4042,4062,4065,4066,4050,4039,4043,4063,4041,4045,4049,4058,4044,4046,4060,4053,4052,4047,4051,4040,4048,4059,4056,4057,4093,4084,4096,4083,4090,4071,4091,4094,4095,4079,4068,4072,4092,4070,4074,4078,4087,4073,4075,4089,4082,4081,4076,4080,4069,4077,4088,4085,4086,5283,5295,5280,5288,5296,5289,5308,5287,5282,5281,5284,5300,5311,5306,5286,5285,5309,5305,5301,5292,5307,5304,5290,5297,5298,5303,5299,5310,5302,5291,5294,5293,42019,39712,1189,1181,1132,1133,1192,1188,1190,1186,1182,1187,1185,1178,1179,1184,1180,1191,1183,39710,39711,681,256,18,28)
Function: update_meta_cache()

#7 @wonderboymusic
11 years ago

  • Keywords has-patch added; reporter-feedback dev-feedback removed
  • Milestone changed from Awaiting Review to 3.7

Here's my take on this:

  1. You never want to arbitrarily update the meta cache for a billion posts at once, you only want to cache data as its read or as a page's bundle of posts is loaded. If you are loading 30,000 posts, chances are, you are only loading them once for something like an import. If your traffic is bursty, you are probably only having a subset of posts read at scale
  2. No function that accepts an arbitrary number of $object_ids that are passed to update_meta_cache() ever reads the return value of the function, so there's no downside to it returning false to skip the billion queries and cache adds
  3. When this function is called with any number of $object_ids greater than the filtered value of meta_cache_limit, we should just bail.

My patch provides the meta_cache_limit filter and sets the default to 1000

#8 @nacin
11 years ago

The limit is not bad but it is tough to get around it when you want to. Either a $force = false parameter, or have update_meta_cache() that wraps a force_update_meta_cache() that ignores any limit.

You have to think that if someone is doing an unbounded query on massive amounts of data (deliberately), they should know to turn off the cache priming.

Or: maybe we should chunk this. If we get greater than 200 IDs, then break them into groups of 100 and do individual queries. That's better than timing out, but at the same time, maybe follows the spirit of cache-priming a bit better than not priming the cache at all.

#9 @nacin
10 years ago

  • Keywords 2nd-opinion added

Not sure what our direction should be here. Any opinions?

#10 @nacin
10 years ago

  • Milestone changed from 3.7 to 3.8

#11 @wonderboymusic
10 years ago

  • Milestone changed from 3.8 to Future Release

#12 @GaryJ
10 years ago

I've got a meal plan CPT, where each meal is linked to multiple recipe CPTs (and that in turn, to multiple ingredient CPTs) with Posts 2 Posts plugin. When trying to show the list of meal plans (edit.php?post_type=mp_mealplan), I'm getting a fatal error due to memory allocation. It points to https://core.trac.wordpress.org/browser/tags/3.9.1/src/wp-includes/meta.php#L747 as being the initial issue (array mapping of intval over all object IDs). Hacking WP to echo a count of $object_ids gives a value of 13847 objects for the instance where the fatal error happens.

There are 2600+ meal plans, each with a guestimate average of 6-10 recipes. Each recipe has 6-10 ingredients on average.

It's on Synthesis hosting, with W3TC (the wp-content/advanced-cache.php references W3TC too).

I can temporarily get around it by bumping the WP memory constants to 512M, but if there was a chunking option, or a way to smooth out or avoid this caching, then it would definitely help the extreme cases.

#13 @jipmoors
9 years ago

It should be made clear (on the codex) that this functionality is not intended to cram up everything you have because of the unloading of cache when the memory is full.

Though I think that batching up makes a lot more sense compared to limiting the number of cache items the system allows. I don't know what kind of system this will run on or how huge the site has become.

Discouraging would be the main goal, but we don't want to break sites or scare people away, I guess.

@jipmoors
9 years ago

Batching meta ids when updating cache

#14 @jipmoors
9 years ago

  • Keywords needs-codex added

#15 @jipmoors
9 years ago

As for the batch size I considered a couple of things; since meta_key is 255 varchar this is a size that we can calculate with, meta_value can be 4GB, so calculating is fairly useless.

Perhaps the memory_limit could be used to set a more dynamic baseline, but by adding the filter option I thought it best to leave it at a relative low value and let caching plugins or users decide for themselves if they want to modify this to minimise the multidimensional overhead added.

Thoughts are welcome of course.

@bmw669
8 years ago

#16 @joehills
8 years ago

I've updated a site to 4.5.1 and am seeing

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/www/html/wp-includes/meta.php on line 830

suddenly on an archive page with ~ 1600 items that worked fine on several older versions of WordPress. Not sure what changed in 4.5.1 to cause this but figured I should let folks know.

#17 @jrfoell
7 years ago

I just want to upvote @jipmoors batch patch. I was getting intermittent 500 errors because of out-of-memory on this very issue. 130 custom post's meta (with apparently a lot of meta) were trying to be cleared. I actually had to reduce the batch count from 250 to 100 to get it to fit in memory (150MB).

Last edited 7 years ago by jrfoell (previous) (diff)

#19 @spacedmonkey
11 months ago

  • Focuses performance added

This ticket was mentioned in PR #6193 on WordPress/wordpress-develop by @pbearne.


3 weeks ago
#20

  • Keywords has-unit-tests added

Implemented batch chunk processing in retrieving and caching non-cached meta values in meta.php. Also added a filter to allow customization of batch size. Corresponding test case has been added to assert the correctness of this batched fetching and caching mechanism in cache.php.

Trac ticket: 14254

#21 @pbearne
3 weeks ago

  • Keywords 2nd-opinion removed
  • Owner set to pbearne
  • Status changed from reopened to assigned

refreshed patch

Note: See TracTickets for help on using tickets.