分片上傳

composeObject方案

總體思路：分別上傳到minio，再通過composeObject()方法合并文件。

public static List<String> createUploadChunkUrlList(String bucketName,String objectMD5,Integer chunkCount){if (null == bucketName){bucketName = chunkBucKet;}if (null == objectMD5){return null;}objectMD5 += "/";if(null == chunkCount || 0 == chunkCount){return null;}List<String> urlList = new ArrayList<>(chunkCount);for (int i = 1; i <= chunkCount; i++){//每個分片作為一個獨立文件上傳String objectName = objectMD5 + i + ".chunk";urlList.add(createUploadUrl(bucketName,objectName,DEFAULT_EXPIRY));}return urlList;}public static boolean composeObject(String chunkBucKetName,String composeBucketName,List<String> chunkNames,String objectName){if (null == chunkBucKetName){chunkBucKetName = chunkBucKet;}List<ComposeSource> sourceObjectList = new ArrayList<>(chunkNames.size());for (String chunk : chunkNames){sourceObjectList.add(ComposeSource.builder().bucket(chunkBucKetName).object(chunk).build());}//合并多個文件到一個文件。在內(nèi)存中合并minioClient.composeObject(ComposeObjectArgs.builder().bucket(composeBucketName).object(objectName).sources(sourceObjectList).build());return true;}

問題：

composeObject()的效率比較低

UploadId方案

步驟：

1、獲取分片的presignedUrl

public Map<String, Object> initMultiPartUpload(String bucketName, String objectName, int totalPart) {Map<String, Object> result = new HashMap<>();try {String uploadId = getMinioClient().initMultiPartUpload(bucketName, null, objectName, null, null);result.put("uploadId", uploadId);List<String> partList = new ArrayList<>();Map<String, String> reqParams = new HashMap<>();reqParams.put("uploadId", uploadId);for (int i = 1; i <= totalPart; i++) {reqParams.put("partNumber", String.valueOf(i));String uploadUrl = minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder().method(Method.PUT).bucket(bucketName).object(objectName).expiry(1, TimeUnit.DAYS).extraQueryParams(reqParams).build());partList.add(uploadUrl);}result.put("uploadUrls", partList);} catch (Exception e) {log.error(StrUtil.format("獲取object[{}]分片 presignedUrl list error。", objectName), e);return null;}return result;}//查詢分片信息public ListPartsResponse listMultipart(String bucketName, String objectName, String uploadId) throws NoSuchAlgorithmException, InsufficientDataException, IOException, InvalidKeyException, ServerException, XmlParserException, ErrorResponseException, InternalException, InvalidResponseException {return getMinioClient().listMultipart(bucketName, null, objectName, null, null, uploadId, null, null);}

2、通過http調(diào)用，PUT數(shù)據(jù)到分片

uploadFile = async () => {console.log("start....")//獲取用戶選擇的文件const file = document.getElementById("upload").files[0];//文件大小(大于5m再分片哦，否則直接走普通文件上傳的邏輯就可以了，這里只實現(xiàn)分片上傳邏輯)const fileSize = file.sizeif (fileSize <= chunkSize){console.log("上傳的文件大于5m才能分片上傳")}//計算當前選擇文件需要的分片數(shù)量const chunkCount = Math.ceil(fileSize / chunkSize)console.log("文件大小：",(file.size / 1024 / 1024) + "Mb","分片數(shù)：",chunkCount)//獲取文件md5const fileMd5 = await getFileMd5(file);console.log("文件md5：",fileMd5)console.log("向后端請求本次分片上傳初始化")//向后端請求本次分片上傳初始化const initUploadParams = JSON.stringify({chunkCount: chunkCount,fileMd5: fileMd5})$.ajax({url: "http://127.0.0.1:7050/api/multi_upload_url/test/m1/upload.test.ui.pdf?totalSize=" + fileSize + "&partSize=5242880", type: 'POST', contentType: "application/json", processData: false, async: false,success: async function (res) {//code = 0 文件在之前已經(jīng)上傳完成，直接走秒傳邏輯；code = 1 文件上傳過，但未完成，走續(xù)傳邏輯;code = 200 則僅需要合并文件if (res.code !== 0) {console.log(res.msg)//composeFile(fileMd5,file.name)return;}console.log("當前文件上傳情況：初次上傳 或 斷點續(xù)傳")const uploadInfo = res.data;uploadId = uploadInfo.uploadId;fileId = uploadInfo.id;const chunkUploadUrls = uploadInfo.parts;//當前為順序上傳方式，若要測試并發(fā)上傳，請將第52行 await 修飾符刪除即可//若使用并發(fā)上傳方式，當前分片上傳完成后打印出來的完成提示是不準確的，但這并不影響最終運行結(jié)果；原因是由ajax請求本身是異步導致的for (item of chunkUploadUrls) {//分片開始位置let start = item.startPosition;//分片結(jié)束位置let end = item.startPosition + item.partSize;//取文件指定范圍內(nèi)的byte，從而得到分片數(shù)據(jù)let _chunkFile = file.slice(start, end)console.log("開始上傳第" + item.partNumber + "個分片")await $.ajax({url: item.url, type: 'PUT', contentType: false, processData: false, data: _chunkFile,success: function (res) {console.log("第" + item.partNumber + "個分片上傳完成")}})}//請求后端合并文件composeFile(fileId)}}) } /*** 請求后端合并文件* @param fileMd5* @param fileName*/ composeFile = (fileId) => {console.log("開始請求后端合并文件")//注意：bucketName請?zhí)顚懩阕约旱拇鎯ν懊Q，如果沒有，就先創(chuàng)建一個寫在這//const composeParams = JSON.stringify({fileMd5: fileMd5,fileName: fileName,bucketName: "img-library"})$.ajax({url: "http://localhost:7050/api/multi_upload_merge/" + fileId, type: 'post', contentType: "application/json", processData: false, success: function (res) {console.log("合并文件完成",res.data)videoPlay(res.data.filePath,res.data.suffix)}}) }/*** 獲取文件MD5* @param file* @returns {Promise<unknown>}*/ getFileMd5 = (file) => {let fileReader = new FileReader()fileReader.readAsBinaryString(file)let spark = new SparkMD5()return new Promise((resolve) => {fileReader.onload = (e) => {spark.appendBinary(e.target.result)resolve(spark.end())}}) }

3、合并分片

public boolean mergeMultipartUpload(String bucketName, String objectName, String uploadId) {try {Part[] parts = new Part[1000];//此方法注意2020.02.04之前的minio服務端有bugListPartsResponse partResult = getMinioClient().listMultipart(bucketName, null, objectName, 1000, 0, uploadId, null, null);int partNumber = 1;for (Part part : partResult.result().partList()) {parts[partNumber - 1] = new Part(partNumber, part.etag());partNumber++;}getMinioClient().mergeMultipartUpload(bucketName, null, objectName, uploadId, parts, null, null);} catch (Exception e) {log.error(StrUtil.format("合并object[{}]分片 error。", objectName), e);return false;}return true;}

4、示例

try {OkHttpClient okHttpClient = new OkHttpClient.Builder().connectTimeout(600, TimeUnit.SECONDS).readTimeout(200, TimeUnit.SECONDS).writeTimeout(600, TimeUnit.SECONDS).build();FileInputStream fi = new FileInputStream("C:\\Users\\lihj\\Desktop\\方案建議 2(1).pptx");FileChannel fc = fi.getChannel();String category = "test";String module = "m2";MultiUploadParamsVo multiUploadParamsVo = new MultiUploadParamsVo();multiUploadParamsVo.setTotalSize((int) fc.size());multiUploadParamsVo.setPartSize(5 * 1024 * 1024);//獲取presignedUrlR<MultiUrlInfo> result = fileServiceFeign.initMultiPartUpload(category, module, "multi_upload.auto.pdf", multiUploadParamsVo, "Y");MultiUrlInfo multiUrlInfo = result.getData();String uploadId = multiUrlInfo.getUploadId();String fileId = multiUrlInfo.getId();//循環(huán)每個分片 上傳文件流for (MultiUrlInfo.MultiPartUrlInfo multiPartUrlInfo : multiUrlInfo.getParts()) {String url = multiPartUrlInfo.getUrl();MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_ONLY, multiPartUrlInfo.getStartPosition(), multiPartUrlInfo.getPartSize());byte[] b = new byte[multiPartUrlInfo.getPartSize()];mbb.get(b);//Request.Body body = Request.Body.encoded(b,null);RequestBody body = RequestBody.create(MediaType.parse("application/octet-stream"), b);Request request = new Request.Builder().url(url).put(body).build();okHttpClient.newCall(request).execute();}//合并分片fileServiceFeign.mergeMultipartUpload(fileId, false, "Y");} catch (Exception ex) {log.error("", ex);throw new RuntimeException(ex);}}

問題

分片上傳文件 size 小于 設定值

io.minio.errors.ErrorResponseException: Your proposed upload is smaller than the minimum allowed object size.at io.minio.MinioClient.execute(MinioClient.java:767)at io.minio.MinioClient.completeMultipartUpload(MinioClient.java:4116)at com.x.cloud.file.util.CloudMinioClient.mergeMultipartUpload(CloudMinioClient.java:74)at com.x.cloud.file.util.MinioTemplate.mergeMultipartUpload(MinioTemplate.java:337)at com.x.cloud.file.util.MinioTemplateTest.initMultiPartUpload(MinioTemplateTest.java:50)

原因：采用分片上傳時，只能并且只有有最后一個分片的size 小于 指定值（默認5M），不然就會報錯。

參考

github：https://github.com/minio/minio

分片上傳：https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-upload-object.html

阿里分片上傳：https://help.aliyun.com/document_detail/84786.html?spm=a2c4g.11186623.2.7.60db62e7bAaXLq

listPart bugs: https://github.com/minio/minio/issues/11408

method:listParts throw io.minio.errors.XmlParserException: null at io.minio.Xml.unmarshal(Xml.java:55)listParts response Initiator->DisplayName is empty服務器端未發(fā)送DisplayName，需要更新minio 服務器端版本（2020.02.04之前的minio服務端有bug）

docker minio：https://registry.hub.docker.com/r/minio/minio

API：https://minio-java.min.io/io/minio/MinioClient.html

minio升級

從 版本： RELEASE.2021-06-17T00-10-46Z 之后，不再僅使用 9000 端口，而是分為API端口（9000）和 控制臺端口（9001，瀏覽器），并且控制界面也不再相同

docker run \-p 9000:9000 \-p 9001:9001 \minio/minio server /data --console-address ":9001"

怎么降低版本

單節(jié)點，可以把文件夾重命名，然后再創(chuàng)建一個目錄，把原來目錄下的所有文件，move到新的目錄下。

總結(jié)

以上是生活随笔為你收集整理的Minio进阶的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。